This is the multi-page printable view of this section. .
PGSTY SILO Blog
- 1: Posts
-
2: Release Notes
- 2.1: SILO Console 2.4.1 Released
- 2.2: silo-pkg 3.14.1 Released
- 2.3: mcli 20260916 Released
- 2.4: mcli 20260913 Released
- 2.5: silo-pkg 3.14.0 Released
- 2.6: SILO Console 2.4.0 Released
- 2.7: Silo Console 2.2.0 Release Notes
- 2.8: Silo Console 2.1.0 Released
- 2.9: Silo Console 2.0.0 Released
- 2.10: silo-pkg 3.13.0 Released
- 2.11: silo-pkg 3.12.0 Released
- 2.12: Silo Pkg 3.11.0 Released
- 2.13: Silo 20260903 Released
- 2.14: Explicit Version Deletes Now Require DeleteObjectVersion
- 2.15: mcli 20260903 Released
- 2.16: mcli 20260806 Released
- 2.17: Silo 20260806 Released
- 2.18: Silo 20260804 Released
- 2.19: mcli 20260804 Released
- 2.20: Silo 20260618 Released
- 2.21: Silo 20260417 Released
- 2.22: Silo 20260325 Released
- 2.23: Silo 20260321 Released
- 2.24: Silo 20260314 Released
- 2.25: Silo 20260214 Released
- 2.26: Silo 20251203 Released
-
3: SILO Security Chronicle
- 3.1: September Security Repairs: Payload Integrity, IAM Revocation and Console Sharing
- 3.2: SN-2026-011: Fix and Release Status
- 3.3: CVE-2025-62506: Session-Policy Privilege Escalation
- 3.4: SILO 20260903 Security Notes: SN-2026-006 through 010
- 3.5: CVE-2026-32285: The jsonparser Advisory That Required No Patch
- 3.6: CVE-2026-33322: OIDC JWT Algorithm Confusion
- 3.7: CVE-2026-33419: LDAP STS Enumeration and the Throttling Chain
- 3.8: CVE-2026-34204: Replication Metadata Injection
- 3.9: CVE-2026-39414: Oversized S3 Select Records and a SIMD Bypass
- 3.10: CVE-2026-40344: Snowball Auto-Extract Authentication Bypass
- 3.11: CVE-2026-41145: Unsigned-Trailer Query Authentication Bypass
- 3.12: CVE-2026-42600: ReadMultiple Storage-REST Path Traversal
- 3.13: Internode Path Containment Audit: Paying Off What CVE-2026-42600 Left Owing
- 3.14: The Parser Knew, the Schema Didn't: Config Keys That Could Take Every Notification Down
- 3.15: Object Grant, Bucket Reach: When 'bucket/*' Could Rewrite the Bucket Itself
- 3.16: Absent Is Not Empty: A Blank versionid and the Fail-Open It Invites
- 3.17: Three Headers, One Lie: Making the Client Source Address Mean Something
- 3.18: Sorted Is Not Increasing: How One Duplicate Part Number Doubled an Object
-
4: Design Records
- 4.1: Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence
- 4.2: An Unsigned Header Is Not Part of the Request
- 4.3: Durable IAM Revocations
- 4.4: Federated CopyObject: Preserve the Destination Contract
- 4.5: Multi-Pool Object Consistency
- 4.6: Object Lock Replication Ordering
- 4.7: SSE-C Replica Integrity
- 4.8: Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation
- 4.9: Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection
- 4.10: SILO Server 20260903 Pre-release Review
- 4.11: Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject
- 4.12: Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts
- 4.13: Conditional DELETE: One Condition, One Logical Object
- 4.14: DSN-Only Database Notifications: A Compatibility Boundary for #53
- 4.15: Preview Text, Never Execute It: SILO Console Text Preview PRD
- 4.16: Go 1.27 TLS Defaults and OIDC Discovery Failure Modes
- 4.17: No I/O Before Auth, No Privilege From Headers
- 4.18: One Endpoint, Two Privileges: Separating User and Group Status
- 4.19: Config Environment Files Are Not Shell Scripts
- 4.20: Two SSE-C Keys, One CopyObject Response
- 4.21: Why CompleteMultipartUpload Must Return ChecksumType: Review of PR #57
- 4.22: When the Total Is Unknown: Folder Download Progress
- 4.23: A ListObjects Shortcut Must Not Turn a Missing Bucket into an Empty One
- 4.24: Read-Only Checksum Audit and Reliable CLI Output
- 4.25: Optional Checksums, Mandatory Failure: Repairing UploadPart and UploadPartCopy Compatibility
- 4.26: BadDigest, InvalidRequest, and the CompleteMultipartUpload Checksum Contract
- 4.27: ListMultipartUploads: Implementation, Upgrade Contract and Design History
1 - Posts
Essays and analysis about MinIO, S3-compatible object storage, and the SILO community fork.
1.1 - MinIO Fork, Promise Kept
Two months ago in “MinIO is Dead, Long Live MinIO,” I promised I’d keep the MinIO fork patched. The recurring objection on HN is fair: can one person actually maintain something like this? The real answer isn’t clicking fork. It’s what happens when CVEs start landing.
Between April 15 and 17, pgsty/minio shipped RELEASE.2026-04-17,
closing four CVEs and a handful of related vulnerabilities disclosed in the same window.
The scope I committed to originally was narrow: no new features, keep the supply chain running, handle reproducible bugs and security issues as they come in. This release is what it looks like when that promise gets tested.
What happened upstream
In December 2025, MinIO moved the open-source repo to maintenance mode. The README said security fixes would be “evaluated case by case.” In February 2026, the repository was archived and the landing page became “this repository is no longer maintained.”
The SECURITY.md in that same archived repo still says: “we will always provide security updates for the latest release.”
Over the past month, four high-severity and two medium-severity vulnerabilities have been disclosed against the final open-source release.
It’s been 184 days since the last upstream release. Vulnerabilities get disclosed; fixes ship only in the commercial build. The guidance for OSS users is a single line: upgrade to AIStor.
AIStor starts around $100k/year for 400 TiB — roughly S3 pricing, for software you install and operate yourself.
It’s a clean arrangement: archive the repo so there’s no obligation to patch, keep publishing CVE advisories for visibility, and route everyone who reads them toward the commercial product.
Someone still has to patch the old one.
What this release fixes
Full write-ups, CVSS arithmetic, and PoCs are in the release notes. The short version:
- CVE-2026-33322 (OIDC JWT algorithm confusion, CVSS 9.8): under certain IdP configurations, an attacker who knows the OIDC ClientSecret can mint a token claiming any identity — including
consoleAdmin— and MinIO will accept it. Vulnerable window: November 2022 through March 2026. About three and a half years. - CVE-2026-33419 (LDAP STS enumeration and brute-force): the login endpoint leaks which usernames are real, and there’s no rate limiting on the subsequent password guessing. End of the chain is an STS credential.
- CVE-2026-34204 (replication-header metadata injection): a regular PUT or COPY with certain
X-Minio-Replication-*headers can write an object into a permanently unreadable state. The data is still on disk; you just can’t read it back out. - CVE-2026-39414 (S3 Select memory exhaustion): one request, one OOM.
- GHSA-hv4r-mvr4-25vw / GHSA-9c4q-hq6p-c237: two signature-verification bypasses on the unsigned-trailer path. Anonymous or forged-signature requests can successfully write objects on certain routes.
Plus the usual dependency cleanup from go-jose, go.opentelemetry.io, and the Go 1.26.2 upgrade itself — about twenty security items in total counting transitive dependencies.

How it got fixed
I said in the earlier post that I’d rely on AI coding agents, and that’s how this round went. My role was closer to “review and decide” than “write code.”
Per-issue flow, roughly:
- Codex drafts first. Given the CVE description and relevant code paths, it produces an initial patch.
- Claude Code reviews adversarially. Picks holes from the attacker’s side.
- Back to Codex. If it agrees with Claude Code’s critique, it reworks. If not, it has to write out why. No silent overrides.
- Another round of review by Claude Code, with both sides’ reasoning on the table. Iterate until they converge.
- Tests. Codex proposes cases, Claude Code adds more, Codex runs them, Claude Code reviews the results.
- I decide. Read the diff, run the tests, merge or send it back with comments.
I didn’t write any of the code in this round. My job was to define the problem, set constraints, pick between approaches, read diffs, run tests, and merge. The GitHub log shows Vonng, Codex, and Claude Code as co-authors — that’s just who did the work.

A few things I noticed about how this runs in practice.
Two heterogeneous agents in opposition catch more than one agent alone. A single agent patching a security bug tends toward confident-sounding fixes that quietly miss a boundary condition. Having a second agent argue against the first filters out most of those.
It forces the tradeoffs into writing. When two implementations diverge, someone has to say why A over B. That exchange is the thing I can actually act on as the person deciding what to merge.
Real maintenance is patch-on-patch, not one-shot. The LDAP STS fix is a good example. The first version landed, and then we realized: successful requests shouldn’t count against the rate limit; X-Forwarded-For shouldn’t be trusted by default; the limiter should key on source IP plus normalized username, not just one. Three follow-up commits before it settled. Iterating through that by hand would have cost a lot more time.
Why this fork exists
Because I use MinIO myself.
MinIO is a production dependency for Pigsty. I need working binaries, a complete console, packages that keep shipping, and someone actually handling CVEs. That keeps the scope narrow. No new features, no turning the repo into a playground. Compatibility, supply chain, fixes when they’re needed.
— Chainguard also ships MinIO container images that track upstream’s post-archive commits, a useful option if you use their images. This fork is a different shape: source tree, RPM/DEB packages, restored console, and doesn’t depend on upstream continuing to push patches somewhere.
The fork is at about 1,300 stars on GitHub and 50,000+ pulls on Docker Hub now. Not remarkable numbers, but enough to tell me I’m not the only one who needed this fork to keep shipping.

If you’re already running OSS MinIO, migration is cheap:
- Docker: swap
minio/minioforpgsty/minio. - RPM / DEB: on GitHub Releases, or via
pig. - Source: pgsty/minio
- Docs: silo.pgsty.com
You don’t need to replace anything around it or relearn the API. In most cases, you’re just pointing a compatible binary at the same deployment. If you want a full HA production setup, Pigsty ships one for free.
Something I use broke; I’m fixing it.
What’s different in 2026 is the cost of “I’m fixing it.” With two coding agents and someone to referee between them, the maintenance load of a mid-sized Go codebase is tractable for one person in a way it wasn’t a year or two ago. That’s about it — not a grand theory about open-source resilience, just the current operating point.
If you’re running OSS MinIO, the migration is cheap and the patches are current. If another CVE drops, I’ll still be here.
1.2 - MinIO Is Dead, Long Live MinIO
MinIO’s open-source repo has been officially archived. No more maintenance. End of an era — but open source doesn’t die that easily.
I created a MinIO fork, restored the admin console, rebuilt the binary distribution pipeline, and brought it back to life.
If you’re running MinIO, swap minio/minio for pgsty/minio.
Everything else stays the same. (CVE fixed, and the console GUI is back)
The Death Certificate
On December 3, 2025, MinIO announced “maintenance mode” on GitHub. I wrote about it in MinIO Is Dead.
On February 12, 2026, MinIO updated the repo status from “maintenance mode” to “no longer maintained”, then officially archived the repository. Read-only. No PRs, no issues, no contributions accepted. A project with 60k stars and over a billion Docker pulls became a digital tombstone.

If December was the clinical death, this February commit was the death certificate.
Today (Feb 14), a widely circulated article titled How MinIO went from open source darling to cautionary tale laid out the full timeline.

Percona founder Peter Zaitsev also raised concerns about open-source infrastructure sustainability on LinkedIn. The consensus in the international community is clear:
MinIO is done.
Looking back at the timeline over the past years, this wasn’t a sudden death. It was a slow, deliberate wind-down:
| Date | Event | Nature |
|---|---|---|
| 2021-05 | Apache 2.0 → AGPL v3 | License change |
| 2022-07 | Legal action against Nutanix | License enforcement |
| 2023-03 | Legal action against Weka | License enforcement |
| 2025-05 | Admin console removed from CE | Feature restriction |
| 2025-10 | Binary/Docker distribution stopped | Supply chain cut |
| 2025-12 | Maintenance mode announced | End-of-life signal |
| 2026-02 | Repo archived, no longer maintained | End of project |
A company that raised $126M at a billion-dollar valuation spent five years methodically dismantling the open-source ecosystem it built.
But Open Source Endures
Normally this is where the story ends — a collective sigh, and everyone moves on.
But I want to tell a different story. Not an obituary — a resurrection.
MinIO Inc. can archive a repo, but they can’t archive the rights that the AGPL grants to the community.
Ironically, AGPL was MinIO’s own choice. They switched from Apache 2.0 to AGPL to use it as leverage in their disputes with Nutanix and Weka — keeping the “open source” label while adding enforcement teeth. But open-source licenses cut both ways — the same license now guarantees the community’s right to fork.
Once code is released under AGPL, the license is irrevocable. You can set a repo to read-only, but you can’t claw back a granted license. That’s the beauty of open-source licensing by design: a company can abandon a project, but it can’t take the code with it.
So — MinIO is dead, but MinIO can live again.
That said, forking is the easy part. Anyone can click the Fork button. The real question isn’t “can we fork it” but “can someone actually maintain it as a production component?”
Why would I do that?
I didn’t set out to take this on. But after MinIO entered maintenance mode, I waited a couple of weeks for someone in the community to step up.
But I didn’t find one. So I did it myself.
Some background: I maintain Pigsty — a batteries-included PostgreSQL distribution with 460+ extensions, cross-built for 14 Linux distros. I also maintain build pipelines for 290 PG extensions, several PG forks, and dozens of Go Projects (Victoria, Prometheus, etc.) packaging across all major platforms. Adding one more to the pipeline was a piece of cake.
I’m not new to MinIO either. Back in 2018, we ran an internal MinIO fork at TanTan (back when it was still Apache 2.0), managing ~25 PB of data — one of the earliest and largest MinIO deployments in China at the time.
More importantly, MinIO is an optional module in Pigsty. Many users run it as the default backup repository for PostgreSQL in production. We did consider several alternatives, but none were a drop-in replacement for MinIO-based workflows.
We use MinIO ourselves, so keeping the supply chain alive was not optional — it had to be done. As early as December 2025, when MinIO announced maintenance mode, I had already built CVE-patched binaries and switched to them.
What We’ve Done
As of today, three things.
1. Restored the Admin Console
This was the change that frustrated the community the most.
In May 2025, MinIO stripped the full admin console from the community edition, leaving behind a bare-bones object browser. User management, bucket policies, access control, lifecycle management — all gone overnight. Want them back? Pay for the enterprise edition. (~$100,000)
We brought it back.

The ironic part: this didn’t even require reverse engineering.
You just revert the minio/console submodule to the previous version.
They swapped a dependency version to replace the full console with a stripped-down one. The code was always there.
We put it back.
2. Rebuilt Binary Distribution
In October 2025, MinIO stopped distributing pre-built binaries and Docker images,
leaving only source code. “Use go install to build it yourself” — that was their answer.
For the vast majority of users, the value of open-source software isn’t just a copy of the source — supply chain stability is what matters. You need a stable artifact you can put in a Dockerfile, an Ansible playbook, or a CI/CD pipeline — not a requirement to install a Go compiler before every deployment.
We rebuilt the distribution:
- Docker Images
pgsty/miniois live on Docker Hub.docker pull pgsty/minioand you’re good.- RPM / DEB Packages
- Built for major Linux distributions, matching the original package specs.
- CI/CD Pipeline
- Fully automated build workflows on GitHub, ensuring ongoing supply chain stability.
If you’re using Docker, just swap minio/minio for pgsty/minio.
For native Linux installs, grab RPM/DEB packages from the GitHub Release page. You can also use pig (the PG extension package manager) for easy installation, or configure the pigsty-infra APT/DNF repo to install from it:
Just works as usual.
3. Restored Community Edition Docs
MinIO’s official documentation was also at risk — links had started redirecting to their commercial product, AIStor.
We forked minio/docs, fixed broken links, restored removed console documentation, and deployed it as the SILO documentation site.
The docs use the same CC Attribution 4.0 license as the original, with necessary maintenance.

Commitments
Some things worth stating up front to set expectations.
No New Features — Just Supply Chain Continuity
MinIO as an S3-compatible object store is already feature-complete. It’s a finished software. It doesn’t need more bells and whistles — it needs a stable, reliable, continuously available build. (I already have PostgreSQL for these, so I don’t need something like S3 table or S3 vector. A stable S3 core is all I need)
What we’re doing: making sure you can get a working, complete MinIO binary, with the admin console included and CVE fixed. RPM, DEB, Docker images — built automatically via CI/CD, drop-in compatible with your existing minio. We keep the existing minio naming and behavior where legally and technically feasible.
This Is a Production Build, Not an Archive
We run these builds ourselves and have been dogfooding them in production for three months. If something breaks, we detect it early and patch it quickly.
I build this primarily for Pigsty and our own usage, but I hope it helps others too.
I’m willing to Track CVEs and Fix Bugs
If you run into issues, feel free to report them at pgsty/minio.
I’ll do my best to fix these — but please don’t treat this as a commercial SLA.
Given that AI coding tools have made bug fixing dramatically cheaper, and that we’re explicitly not adding any new features, I believe the maintenance workload is manageable. (how often do you see one?)
Trademark Is Tricky, But We’ll Cross That Bridge When We Come to It
Disclaimer
Trademark Notice: MinIO® is a registered trademark of MinIO, Inc. This project (pgsty/minio) is an independently maintained community fork under the AGPL license. It has no affiliation with, endorsement by, or connection to MinIO, Inc. Use of “MinIO” in this post refers solely to the open-source software project itself and implies no commercial association.
AGPLv3 gives us clear rights to fork and distribute, but trademark law is a separate domain. We’ve marked this clearly everywhere as an independent community-maintained build.
If MinIO Inc. raises trademark concerns, we’ll cooperate and rename (probably something like silo or stow).
Until then, we think descriptive use of the original name in an AGPL fork is reasonable — and renaming all the minio references doesn’t serve users.
AI Changed the Game
You might ask: can one person really maintain this?
It’s 2026. Things are different now.
AI coding tools are changing the economics of open-source maintenance.
With tools like Claude Code & Codex, the cost of locating and fixing bugs in a complex Go project has dropped by more than an order of magnitude. What used to require a dedicated team to maintain a complex infra project can now be handled by one experienced engineer with an AI copilot.
Maintaining a MinIO build without adding new features is a manageable task. The key requirement is testing and validation. and we already have that scenario, which lets us verify compatibility, reliability, and security in practice.
Consider: Elon cut X/Twitter’s engineering team down to ~30 people and the system still runs. Maintaining a MinIO fork without new features is considerably less daunting
Just Fork It
MinIO Inc. can archive a GitHub repo, but they can’t archive the demand behind 60k stars, or the dependency graph behind a billion Docker pulls. That demand doesn’t disappear — it just finds its way out.
HashiCorp’s Terraform got forked into OpenTofu, and it’s doing fine. MinIO’s situation is actually more favorable —
AGPL is more permissive for forks than BSL, with no legal gray area for community forks.
A company can abandon a project, but open-source licenses are specifically designed so the code can’t die.
Fork is the most powerful spell in open source. When a company decides to shut the door, the community only needs two words:
Fork it.
Reference
- MinIO Is Dead
- MinIO Is Dead, Are There Alternatives?
- From AGPL to Apache: Reflections on Pigsty’s License Change
- MinIO: Promise made, Promise kept
1.3 - MinIO Is Dead. Which Next?
MinIO announced maintenance mode two days ago. I ranted in “MinIO Is Dead” and immediately got flooded with “so what now?”
The usual suspects: Ceph, RustFS, SeaweedFS, Garage. I packaged all of them for Linux (RPM/DEB) and ran them through the grinder.
Short version: there’s no perfect substitute. Ceph is powerful but overkill; SeaweedFS rocks tiny files but needs an external metadata DB; Garage is cute but too barebones; RustFS targets the MinIO niche but is still alpha.
Quick scan of the field
MinIO is the open-source S3 clone. If all you need is basic object CRUD, any S3-compatible store works. But parity with MinIO means more than APIs—it’s about reliability, operability, tooling, documentation, SOPs. Replacing it cleanly is hard.
Ignoring commercial clouds, here’s the OSS menu:
- Ceph – arguably the best choice for enterprises, but brutally complex. Most folks don’t need block + file + object in one, and it requires extras like Podmon. MinIO’s single binary spoiled us.
- SeaweedFS – optimized for oceans of small files; O(1) disk seeks make it absurdly fast there. But it relies on an external metadata store. If you want a general-purpose object store, that dependency is annoying.
- Garage – built by Deuxfleurs with NGI funding. Delightfully light (10 MB), great for self-hosters and edge nodes. But S3 compatibility is thin: no versioning, no cross-region replication, no IAM. Enterprises will laugh.
- RustFS – the only project explicitly chasing “drop-in MinIO,” but it’s still alpha.
RustFS vs. MinIO
RustFS looked the most promising, so I wired it into Pigsty as a MinIO replacement. Most logic carried over, but a few differences popped up:
- Certificates must follow specific naming rules.
- Health checks differ from MinIO’s endpoints.
mc admindoesn’t work; you can’t push fine-grained IAM policies. That’s a deal-breaker for many teams.
It ran, but I’m not shipping alpha software into production, so I shelved the branch. I’ll revisit when RustFS hits GA.
Will RustFS repeat MinIO’s mistakes?
RustFS has potential, but I worry it’ll retrace MinIO’s path. I asked the AI big three (GPT‑5 Pro, Claude 4 Opus, Gemini 3 Pro) to audit the project. Gemini leveled some serious accusations; Claude corroborated.
The red flags match MinIO’s history: Apache 2.0 license + copyright assignment CLA + single commercial gatekeeper. With that risk profile, I’m downgrading RustFS from “optimistic” to “cautious wait-and-see.”
So what now?
Pigsty bundles MinIO as an optional module for PostgreSQL backups or as an on-prem S3 for apps like Supabase. After surveying the alternatives, I’m not eager to swap it out. I might add a pgBackRest-native backup server option, but ripping out MinIO today feels premature.
Best plan: stay on the latest MinIO release, lock the version, isolate it on the network, and wait a few months. Maybe the community forks it; maybe RustFS matures. Adjust when reality changes.
RustFS still has a golden window to seize MinIO’s niche with a safer, community-friendly fork. That window is measured in months, not years.
If you stick with MinIO
Use the latest build, not the April 22, 2025 edition with the GUI. There’s a serious CVE in the interim:
- CVE-2025-62506 – privilege escalation via session-policy bypass (HIGH). Low-privilege users can mint new accounts and escalate.
In a locked-down intranet the risk is manageable, but you still want the fix, which landed in the 2025‑10‑15 release. MinIO pulled the prebuilt binaries starting with that version, offering source only. Annoying, but it’s Go—go build and you’re done. I forked MinIO, ran their packager, and produced RPM/DEBs for 2025‑12‑03 so I’m not deploying vulnerable bits: https://github.com/pgsty/minio

Security patches still need humans. MinIO claims they’ll fix critical issues, but if the community wants a maintained fork, now’s the moment. Start from 2025‑04‑22, cherry-pick critical bug/security fixes, and keep a community LTS alive.
MinIO is “done” software. It doesn’t need the latest S3 gimmick (Vector/Table); it needs steady bugfixes. That’s perfect for a community branch. Plenty of storage vendors rely on MinIO; maintaining a fork beats writing a new object store from scratch.
2026-02-14 Update: MinIO’s official repo has been fully archived and is no longer maintained. Besides, I’ve personally maintained an oss fork of minio:
pgsty/minio/ Docs: https://silo.pgsty.com. Which based on the last upstream version 2025-12-03 with restored console capabilities.
1.4 - MinIO is Dead
December 3, 2025 was a day to mark in open-source software history. MinIO’s team updated the project status on GitHub, announcing the MinIO open-source project was entering “maintenance mode.” This basically declared the death of MinIO as an open-source project.
MinIO the company has finally completed its transformation from a dragon-slaying hero into the very dragon it once sought to slay.

From Dragon-Slayer to Dragon
Democratization Era (2014–2019): The Apache of Object Storage
MinIO was founded in 2014 with a highly idealistic vision – to be “the Apache of object storage.” In an era dominated by AWS S3, MinIO’s ultra-lightweight design (a single static binary) and 100% S3 API compatibility quickly won developers’ hearts.
During this phase, MinIO was licensed under the liberal Apache 2.0 license, encouraging developers to integrate it into all kinds of applications. Its core pitch: “turn any hardware into AWS S3.” This open strategy was wildly successful. MinIO claimed its Docker image had been pulled over 1 billion times, making it the world’s most widely deployed object storage service. At this point, MinIO was a darling of the cloud-native stack – the default storage backend in many Kubernetes setups.
License Weaponization (2019–2025): The AGPL War
The first major crack in community relations appeared around 2019–2021. MinIO announced it was changing its core license from Apache 2.0 to GNU AGPLv3.
The official explanation was that this move aimed to prevent cloud providers (like AWS, Azure) from “freeloading” the code and repackaging it as proprietary services — a common defensive tactic in open source. During this period, MinIO shifted from being a community guardian to an aggressive defender of its IP. In 2022, MinIO publicly accused Nutanix Objects of violating its license and revoked Nutanix’s right to use MinIO; in 2023, MinIO sued high-performance filesystem vendor Weka on similar grounds. These legal actions, though legally contentious, sent a clear signal: MinIO no longer welcomed commercial use without paying up. This set the legal and psychological stage for the full lockdown that would come in 2025.
Control Plane Neutered (May 2025)
In May 2025, MinIO decided to strip the MinIO Console out of the community edition. The console was a critical GUI for bucket management, IAM, monitoring, and audit logging. After this removal, the open-source MinIO was left with only a basic “object browser” GUI – essentially just a file viewer/downloader.
Meanwhile, key admin features like policy management, site replication configuration, and lifecycle management were moved entirely into the commercial enterprise edition. This change downgraded the open-source MinIO from a full-featured storage management system into a mere data-plane component, robbing it of the control-plane capabilities needed to run as a standalone product in production.
Cutting Off Binary Distribution (Oct 2025)
On October 15, 2025 – right as a critical security vulnerability (CVE-2025-10-15T17-29-55Z / GHSA-jjjj-jwhf-8rgr) was disclosed – MinIO stopped publishing updated Docker images to Docker Hub and Quay.io. The timing of this move was highly strategic. By cutting off binaries during a major security incident, MinIO effectively used security as a bargaining chip.
This decision directly broke the automated deployment pipelines for countless users. Helm charts, Ansible playbooks, and Terraform scripts expecting minio/minio (or Bitnami’s minio) image suddenly failed to find updates.
Auto-scaling groups trying to pull new nodes hung due to missing images. For teams without a Go build environment or an internal container registry, MinIO instantly became unusable.
Maintenance Mode (Dec 2025)
On December 3, 2025, MinIO, Inc. officially updated its channels and GitHub repo to announce that the open-source project is now in “maintenance mode.” The README stated that there will be no further feature additions or improvements, issues and PRs will no longer be reviewed, and even critical security fixes would be provided “as appropriate.” No more RPM/DEB packages or Docker images will be released. Essentially, anyone needing updates or support is advised to switch to the commercial AIStor product.

Technical Impact: Damage to the Open-Source Ecosystem
MinIO’s move to maintenance mode dealt an immediate and far-reaching blow to many tech stacks.
Broken CI/CD Pipelines and an Automation Crisis
Thousands of Helm charts, Ansible playbooks, and Terraform scripts depend on the minio/minio (or Bitnami’s minio) container image.
With official images no longer published, third-party packagers like Bitnami — who can’t get a stable upstream release — also had to stop updates.
- Cascade effect: Deployments in fresh environments started failing outright. Auto-scaling groups, upon launching new instances, would hang or error out when the MinIO image couldn’t be pulled.
- Cost of fixes: Companies now have to rewrite their deployment scripts to point to a self-hosted image, and set up internal build pipelines to compile and package MinIO from source.
Security Vacuum: CVE Patches Go Private
The most lethal consequence of halting binary distribution is delayed security patches. In the October 2025 incident, for example, MinIO effectively withheld the patched binaries for the vulnerability.
- Risk exposure: Companies without dedicated security teams are forced to keep running older, vulnerable versions with known critical flaws.
- Compliance nightmare: For organizations under PCI-DSS, HIPAA, SOC2, etc., not being able to obtain vendor-signed security updates is a compliance disaster. Lacking official patches, they technically fall out of compliance.
Exponentially Higher Ops Complexity
Removing the UI wasn’t just a hit to user experience – it increased operational burden.
Tasks that used to be a few clicks in the Console (configuring bucket policies, setting user permissions) now require ops engineers to master the mc CLI or hand-craft complex JSON policy docs.
This raises the skill floor and makes MinIO far less friendly as a lightweight internal tool.
Underlying Reasons: Pressure from Capital and Commercialization
The driving force behind MinIO’s decisions is the logic of venture capital. By 2025, MinIO had raised a total of $126 million in funding. The most significant was a $103 million Series B in January 2022 led by Intel Capital, SoftBank Vision Fund II, and General Catalyst, which crowned MinIO a unicorn (valued over $1 billion).
In VC terms, a $1B valuation means the company must show a clear path to IPO — typically demanding $100M+ in Annual Recurring Revenue (ARR) and rapid growth. In Feb 2025, MinIO announced its ARR had grown 149% over the past two years businesswire.com. Impressive growth, but to live up to a sky-high valuation, organic conversion alone wasn’t enough.
Cutting off the free open-source offering is the most direct way to force a huge user base into paid customers.
In 2025, MinIO underwent a full rebrand and launched “MinIO AIStor,” styling itself as “the data backbone for enterprise AI.” Management recognized that general-purpose object storage (for backups, file servers, etc.) was a red-ocean market with thin margins, whereas generative AI’s appetite for high-throughput data (the exascale AI era) promised the next big surge. By tuning its product for AI workloads and focusing on Fortune 500 enterprises linkedin.com, MinIO essentially decided to cut loose its low-value open-source user base. The move to maintenance mode signaled MinIO’s official pivot from a broad open-source project into a vertical, high-end AI software vendor.
MinIO isn’t a garage hobby project by a few geeks anymore; it’s a company that took $126M in VC and is valued at over $1B. Backed by Intel Capital and SoftBank, once you take that money, your boss is no longer the users — it’s the investors. And what do investors want? ARR, growth, IPO. You tell them, “We have a billion Docker pulls!” and they’ll ask, “How many dimes did those pulls pay us?”
The reality is brutal. To the VCs, those small businesses and individual devs using free MinIO are low-value assets. They open issues and ask for support — consuming expensive engineer time, bandwidth, and servers — yet will never convert to paying customers. MinIO’s leadership knows their real cash cows are the Fortune 500 firms doing generative AI. The ones training GPT models or running self-driving pipelines need AIStor, ultra-high performance, and 24/7 enterprise SLAs.
So flipping the project into “maintenance mode” is essentially an asset carve-out. MinIO is cutting away the “dead weight” (free users) and concentrating on the milkable “cash cows” (enterprise AI clients). In business strategy this is called focus. To the investors, it’s being responsible. But from the perspective of open source, it’s simply betrayal.
Personal Reflections
I started using MinIO around 2018 (back when it was Apache-licensed). We built a few multi-petabyte object storage clusters for videos, images, backups — probably one of the largest MinIO deployments in China at the time. I wrote deployment/monitoring playbooks for MinIO (still open-sourced in Pigsty).
As an open-source startup founder, I can understand the motivation behind these moves. But as an open-source contributor and user — I also know many folks right now have one phrase in their minds: “I have never seen such shamelessness.”
An open-source license isn’t a shackle, but it is a social contract. Developers contribute code, users contribute testing, feedback, and reputation; together, they make a project successful. MinIO enjoyed a decade of community goodwill and parlayed the bragging rights of “#1 in global downloads” into venture funding. Then it turned around and told the very users who propped it up: “You free-riders, get lost.” This kind of move breaks the fundamental trust that open source is built on.
This “bait and switch” tactic is even more nauseating than a crypto rug pull. A rug pull only takes your money — MinIO is pulling the rug out from under the tech stacks of thousands of companies. Adopting a technology isn’t just picking up a binary; it’s buying into an ecosystem and a design philosophy. They got everyone onboard, let the switching costs pile up sky-high, and then suddenly kicked away the ladder. In fact, as open-source expert Tison thoroughly discussed in his article The Bait-and-Switch Open-Source Strategy, the core issue with this model is deception.
MinIO betrayed the community, so the community may abandon it as well. Alternatives like Garage, SeaweedFS, or the new RustFS are ready to step in.
If I have to sum up my feelings, I’d borrow a line from The Hitchhiker’s Guide to the Galaxy:
—— “So long, and thanks for all the fish.”
2026-02-14 Update: MinIO’s official repo has been fully archived and is no longer maintained. Besides, I’ve personally maintained an oss fork of minio:
pgsty/minio/ Docs: https://silo.pgsty.com. Which based on the last upstream version 2025-12-03 with restored console capabilities.
2 - Release Notes
Each published SILO version has its own page with the release date, major changes, security fixes, dependency updates, and related commits.
2.1 - SILO Console 2.4.1 Released
SILO Console v2.4.1
was released on September 16, 2026 from
1360e26d976d.
It integrates pkg v3.14.1 and mcli 20260916, restricts the anonymous sharing proxy,
streams multi-object ZIP downloads and adds signed, traceable release artifacts.
Shared downloads
The fix for #52 restricts the anonymous proxy to object-content GET requests at the configured S3 origin. It rejects redirects, system paths and query-selected operations that are not object downloads. Public objects, presigned links and version-specific downloads continue to work. No new sharing-disable environment variable is needed.
Thanks to Jiri Pejchal (@jiri-pejchal) for reporting the internal-metrics exposure. See the sharing boundary for the supported request forms and reverse-proxy requirements.
Password-policy migration
This patch release includes an authorization change. The Change Password
button and session capability use admin:ChangeMyPassword; creating users and
resetting another user’s password continue to use admin:CreateUser.
With the matching SILO Server, denying CreateUser alone no longer locks the
caller’s own password. To retain that restriction, deny both actions in the
same statement before upgrading, keeping its original scope and conditions.
The updated built-in readonly permits self-service password changes and no
longer overrides a separate CreateUser Allow. Saved policy overrides retain
their stored statements. Upgrade Server, Console and pkg together and review
the migration and rollback guide.
Downloads and browser recovery
- Multi-selection ZIP downloads stream to a file writer when the browser supports it, with backpressure, cancellation and duplicate-transfer protection. Other browsers hand the download to their native download manager. Neither path buffers the complete ZIP in JavaScript.
- Progress uses a percentage only when the response supplies a reliable total. The native fallback identifies the handoff explicitly; cancellation then belongs to the browser’s download manager. Selections over 5 GiB or with an unknown total recommend mcli for long-running transfers.
- Invalid routes and rendering failures offer recovery without clearing preferences. English and Chinese labels, icon controls, sign-out text and keyboard-accessible tooltips are completed.
Dependencies and embedding
| Component | Selected version |
|---|---|
| Shared package | github.com/pgsty/silo-pkg/v3 v3.14.1 |
| MC source | github.com/pgsty/mc v0.0.0-20260916070421-e952aa78f10a — mcli 20260916 |
| Upstream minio-go | v7.3.1-0.20260915093545-32e1f32cb176 |
| JWX / strfmt | v3.3.0 / v0.27.2 |
| React Router | v7.18.4 |
| Build tools | Go 1.27.1, Node 24.21.0, Yarn 4.13.0 |
The SDK handles S3 errors embedded in HTTP 200 CopyObject responses; JWX fixes custom JSON field-name escaping and strfmt updates hostname validation for Go 1.27. The embedded frontend and third-party credits are regenerated from this dependency graph. go-systemd v22.6.0 and tablewriter v0.0.5 remain intentional compatibility pins.
Go embedders select Console as
github.com/pgsty/silo-console v0.0.0-20260916075814-1360e26d976d through the
historical github.com/minio/console module replacement. Go does not inherit
dependency replacements: copy the explicit MC replacement from the
tagged README.
The supported administration target is the coordinated SILO stack; upstream
MinIO/MC compatibility is best effort.
Packages and verification
The official container image is on Docker Hub, with Linux amd64 and arm64 support:
The release contains 44 assets: binaries and bundles, DEB/RPM/APK packages,
source, legal notices, SPDX SBOMs, a checksum manifest and its Sigstore bundle.
Checksums and the OCI image use keyless Cosign signatures; GitHub provenance
records the build workflow and tagged release inputs. The image’s latest
tag advances only after the published release passes signature, provenance,
SBOM and anonymous-access checks. See the
release contract
for online and offline verification.
Linux packages retain minio-console.service, console-user and
/etc/default/console. The service now uses /var/lib/silo-console for state,
/etc/silo-console/certs for certificates, and a 90-second shutdown limit.
Before restarting an existing installation, migrate certificates with the
documented ownership or retain the old path through CONSOLE_OPTS. The installer
preserves old certificates and keys and does not restart the service for you.
See the package upgrade instructions.
The exact release source passed the complete CI matrix, vulnerability checks and release verification. Local validation also covered 251 frontend unit tests, 51 browser tests, byte-identical embedded assets and sharing in standalone and embedded deployments. The downloaded Darwin/arm64 executable was independently checked against the signed checksum manifest and exercised against a SILO test server.
2.2 - silo-pkg 3.14.1 Released
v3.14.1 was published from
fa657ef4.
This is a dependency and tooling release of github.com/pgsty/silo-pkg/v3;
the diff from v3.14.0
changes no package Go source or public API.
Dependency changes
- JWX advances from v3.2.0 to v3.3.0. The vendor’s release notes
identify GHSA-4cf7-xm37-g63h: custom JSON field names must be escaped during
encoding. Reachability depends on accepting attacker-controlled custom claim
names. The package’s
envJWT path uses registered claim names; dependency presence alone does not establish an exploitable mcli path. - Upstream minio-go advances to
v7.3.1-0.20260915093545-32e1f32cb176, including minio-go #2306: recognize an S3CopyObjecterror embedded in an HTTP 200 response instead of reporting a successful copy. The fix applies to the SDK’s CopyObject path; it does not establish every application’s selection of that path. - Testify advances to v1.12.1 and the lint configuration adopts
gomodguard_v2. The Go 1.26 floor, Go 1.27.1 toolchain and go-systemd NetBSD replacement remain.
Component and migration boundary
The v3.14.0 password-policy change remains
in force; this patch does not revert it. Preserve the paired
admin:CreateUser / admin:ChangeMyPassword Deny where that was the intended
restriction, using the migration guide.
mcli 20260916 and Console 2.4.1 select this package. The Server source baseline reviewed on September 16 still selects v3.14.0; publishing a module does not update a compiled Server. The component matrix separates published components from the next Server’s selected dependencies.
Release verification is recorded in the linked release and PR #9. Those records are upstream release evidence, not a new execution of their tests by this documentation update.
2.3 - mcli 20260916 Released
Release: RELEASE.2026-09-16T00-00-00Z · Package: 20260916000000.0.0 · Source: e952aa78f10a.
Changes since 20260913
- Adopt silo-pkg v3.14.1, upstream minio-go v7.3.1-0.20260915093545-32e1f32cb176, and JWX v3.3.0. CopyObject retries and propagates S3 errors embedded in HTTP 200 responses, so a failed copy cannot authorize
mvto delete its source. JWX correctly escapes JSON field names. - Report
mirrorpermission failures, including unreadable sources, rejected writes and failed local destination removals. Later objects continue; a finite mirror with failures exits 1. These per-object permission errors do not require--skip-errorsand do not restart an entire watch scan. - Preserve the existing cancellation/retry path for listing and watcher failures. Persistent listing permission denial can still cause repeated scans; this release does not redesign that retry policy.
- Suppress normal final success statistics after a failed mirror. Explicit
--summaryretains statistics: JSON usesstatus: failure; text error diagnostics remain on stderr. Transfers completed within one clock tick no longer produce infinite JSON speeds. - Return failure for legalhold set/clear errors, partial recursive retention failures, and moves whose source deletion fails.
mvwaits for deletion results; successful copies are not rolled back. Retention reports each failed object once instead of duplicate or misleading diagnostics. - Reject empty retention durations and invalid
find --regexexpressions with normal CLI errors instead of panics.
Automation and upgrade notes
Affected failure paths that returned 0 now return 1. Check final exit status and error records. Per-object success start messages and progress byte counters are not proof that a transfer completed. For mv, a deletion failure may leave both the source and the successfully copied destination; inspect both before retrying.
Configuration format, aliases, MC_* variables, commands and package naming remain unchanged. The password-policy split introduced by pkg v3.14.0 is unchanged in v3.14.1. Upgrading only mcli does not update an installed Server or Console, alter Server authorization, or replace a client bundled in an existing Server image. See the component matrix and password-policy migration.
SILO is the maintained administrative integration target. Compatibility with unmodified upstream MinIO and other S3 services remains best effort.
Delivery and validation
The release contains 19 assets: six Linux/macOS/Windows archives for amd64/arm64, six Linux RPM/DEB/APK packages, one archive checksum manifest and six package checksum sidecars. Archives, the manifest and DEB/APK packages have verified Sigstore build attestations bound to this signed tag and source commit. RPMs carry the PGSTY GPG signature (9592A7BC7A682E7333376E09E7935D8DB9BD8B20).
The final release commit passed Go tests on Linux, macOS and Windows with the race detector, Linux 386 tests, cross-compilation, vulnerability scanning and the release packaging pipeline. Pre-release acceptance also exercised both the published SILO 20260903 and fixed Server source 3c26a8b0b5bd, Object Lock and permission failures, CopyObject error injection, and both-architecture RPM/DEB/APK installation and upgrades. No reachable or imported-package vulnerability was found; an unused OpenPGP module-only advisory remains.
CI: VulnCheck · Test Release Pipeline · Crosscompile · Go · Release · Docker
The docker.io/pgsty/mc release and latest tags resolve to the verified Linux amd64/arm64 manifest:
Download and installation · Client compatibility · Changes since 20260913.
The HTTP-200 CopyObject repair applies when mcli selects a single CopyObject: objects smaller than 64 MiB or explicit --disable-multipart. Larger copies normally use ComposeObject; this fix does not substitute for separate evidence for that path.
See pkg 3.14.1 for the precise JWX/GHSA-4cf7-xm37-g63h scope. A failed --json mirror without --summary emits no trailing statistics record; --skip-errors does not turn recorded failures back into exit 0. Some retention JSON failure records carry status: failure with an error object serialized as {}; consumers must not require a human-readable message.
2.4 - mcli 20260913 Released
Release: RELEASE.2026-09-13T00-00-00Z
· Package: 20260913000000.0.0
· Source: 4f609a4d.
Changes since 20260903
- Preserve historical destination versions under
mirror --remove --watch. - Keep service-restart dry runs from executing a restart and make noninteractive behavior explicit.
- Return failure for transfer and S3 Select errors, honor explicit checksums on
empty uploads, and retain
pipeJSON output when quiet mode is enabled. - Accept on/off boolean environment values and repair CLI/JSON behavior across supported platforms.
- Adopt pkg v3.14.0 and upstream SDK
v7.3.1-0.20260910142817-60bd07042d49, refresh Go x/* and UBI dependencies, and keep Go 1.27.1. NetBSD retains the go-systemd v22.6.0 portability pin.
The package retains the policy Deny/NotResource and bounded wildcard repairs from pkg v3.13.3. Correctly preserved denies may now reject requests that relied on lost clauses. Recover already-lost clauses from the original policy source.
The new pkg also changes password capabilities. Read the password-policy migration before a coordinated Server upgrade. This mcli release alone does not change an installed Server’s action mapping. September 16 follow-up: the matching password UI is released in Console v2.4.1; see the component matrix for the coordinated upgrade.
Delivery and validation
The immutable release contains 19 assets: six archives for Linux, macOS and Windows on amd64/arm64, RPM/DEB/APK packages for both Linux architectures, checksums and package checksum sidecars. Archives and the checksum manifest have verified build attestations; RPMs carry the PGSTY GPG signature. Native macOS arm64 version and Go build metadata were checked against the released pins.
The release and latest tags of docker.io/pgsty/mc resolve to the same verified
amd64/arm64 manifest:
Container publication completed; both architecture images were pulled and their version output checked. Go, cross-compilation, vulnerability and Test Release gates passed before tagging. No reachable or imported vulnerable package was reported; unused OpenPGP retains module-only GO-2026-5932. Compatibility with upstream MinIO and other S3 endpoints is best effort; SILO is the maintained administrative integration target.
Download and installation · Compatibility notes · Complete source diff.
pipe streams nonempty stdin until EOF; empty input uses a regular zero-byte PUT instead of unknown-size multipart. Explicit checksum options reach the upload path. Quiet/non-TTY execution keeps progress disabled while retaining JSON results; that output correction is not limited to empty input.
2.5 - silo-pkg 3.14.0 Released
v3.14.0 was published on
2026-09-13 from 827f8109.
It is consumed directly as github.com/pgsty/silo-pkg/v3.
Password-policy migration
This release changes authorization semantics. The public
Policy.IsAllowedActions method now reports admin:ChangeMyPassword unless
explicitly denied; admin:CreateUser requires an explicit Allow. With matching
Server/Console source, a CreateUser deny alone no longer locks the caller’s
password, while a ChangeMyPassword deny does.
The built-in readonly drops its CreateUser deny. A separate CreateUser Allow
can therefore become effective; the added consolereadonly follows the same
split and includes bucket listing. Neither policy grants user administration
by itself. Saved policies and overrides are not rewritten.
To preserve the old combined restriction, retain both actions in the same Deny statement, including its original scope and conditions, before upgrading and throughout rollback. Old Servers do not enforce a password-only deny for this endpoint. Read the full migration guide.
September 16 follow-up: Console v2.4.1 ships the matching password UI with pkg v3.14.1 and mcli 20260916. Follow the component matrix for the coordinated upgrade.
Dependencies and verification
- Upstream minio-go is pinned to
v7.3.1-0.20260910142817-60bd07042d49: configurable upload limits, streaming Content-Type signing, caller TLS trust for RDMA, consistent listing checksums and optional restore status. - Go x/* dependencies were refreshed, with govulncheck 1.8.0. The library retains its Go 1.26 floor and Go 1.27.1 toolchain; public Go signatures are unchanged.
- The go-systemd v22.6.0 replacement remains necessary for NetBSD compilation.
- Full race suites passed on Go 1.26.8 and 1.27.1, along with lint and LDAP configuration validation. Vulnerability scanning found no reachable or imported vulnerable package; unused OpenPGP code retains module-only GO-2026-5932. The release resolves through the Go proxy and checksum database.
The earlier v3.13.3 policy Deny/NotResource and bounded wildcard fixes remain included. Policies whose clauses were already lost must be recovered from their original source. The module-path migration introduced in v3.13.0 is complete in all four maintained components.
2.6 - SILO Console 2.4.0 Released
v2.4.0 was published
on 2026-09-08 from c103d08e.
September 16 follow-up: Console v2.4.1 is the
current release. This article records the v2.4.0 behavior and dependencies.
Released behavior
- Object-browser directories load 100 entries per page by default, with choices of 50, 100, 250, 500 and 1000. Each page uses one S3 continuation-token request. Sorting, filtering and selection apply to the displayed page, not a complete directory scan; there is no unbounded “all” mode.
- Version deletion stays bound to the selected object and directory deletion to its prefix. Lifecycle edits preserve independent actions and unexposed settings; replication rule deletion is saved consistently.
- Transient network failures preserve sessions. Empty versioning state, malformed sidebar preferences and stale dropdown selections are handled.
- Embedded Console restores loopback proxy behavior when Server explicitly enables the policy; standalone trust defaults remain unchanged.
- Dependencies include pkg v3.13.3, MC source
v0.0.0-20260908140805-c8aa5d25a63a, and upstream SDK0e78d3f18efe. This release consumed MC as source and did not itself publish a new mcli tag.
Correctly preserved Deny/NotResource clauses may reject requests that relied on old deduplication defects. Restore already-lost clauses from the original policy source. Embedders must copy the MC replacement from the tagged README.
Later release
Console v2.4.1 adds restricted shared downloads, the password-permission split, streaming multi-object ZIPs, browser recovery and verifiable release artifacts. It selects pkg v3.14.1 and mcli 20260916. v2.4.0 uses the earlier browser-memory ZIP path. Read the password migration before upgrading.
The exact v2.4.0 source passed the complete Console CI matrix. The authoritative historical record is its tagged changelog, object-browser guide, and release assets. Later signature/provenance requirements do not retroactively apply to old assets.
2.7 - Silo Console 2.2.0 Release Notes
Version: v2.2.0 · Release commit: 7dc4258a6 · Status: released · Repository: pgsty/silo-console
SILO Console 2.2.0 is a correctness-and-hardening release. It adds one new feature — a strictly bounded text preview for logs and structured text — and spends the rest of its budget making existing surfaces tell the truth: downloads that cannot silently ship a truncated archive, progress bars that cannot fabricate a percentage, permission gates that actually disable what they claim to disable, a user API that no longer entangles status changes with group membership, and database notification forms that emit exactly the connection string the server will store.
Underneath, the dependency stack moves to Go 1.27.0 and onto the maintained SILO forks: pgsty/silo-pkg 3.12.1 replaces upstream minio/pkg, pgsty/mc replaces upstream mc as a library, minio-go moves to 7.3.0, and the etcd client line moves to 3.7.1, closing CVE-2026-73500.
The final change set since v2.1.1 is 33 commits touching 176 source files (+8,794/−2,057 lines, not counting the regenerated embedded frontend assets).
Why 2.2.0 and not 2.1.2: the user-visible fixes alone would justify a patch release, but this cycle changes the provider of the shared policy/certificate implementation, crosses an etcd client minor boundary, splits a REST route, and deliberately changes download-failure semantics. Each of those deserves a minor-version line in the compatibility notes below rather than a silent patch.
Text Preview for Logs and Structured Text
The object browser can now preview .log, .txt, .json, and .xml objects as literal text. The implementation is deliberately paranoid, because “render arbitrary bucket content in the admin UI” is an XSS invitation:
- Bounded by construction. The request carries
Range: bytes=0-1048576; the response is streamed into a fixed buffer with a hard cap, andContent-Range/Content-Lengthare strictly validated against what was actually read. An object over 1 MiB reports too large — it is never partially rendered as if complete. - Text or nothing. The bytes must decode as strict UTF-8 (
TextDecoderwithfatal: true) and must not contain NUL; anything else reports not previewable instead of rendering mojibake or binary junk. Eligibility is an exact allowlist — the four extensions plus the precise MIME typestext/plain,application/json,application/xml,text/xml— and.html/.htm/.xhtmlare explicitly excluded even when their metadata claimstext/plain. - No active document. Content renders into a single DOM text node — never
innerHTML, never an iframe, never a JSON-to-HTML transform. A regression suite feeds it HTML, SVG, and XML payloads and asserts they stay inert text. - Race-proof. Switching quickly between objects invalidates in-flight previews by generation, so a slow response for the previous object cannot paint over the current one. Cancellation and retry are first-class states.
- Anonymous-friendly. Public/anonymous object pages get the same preview through the anonymous request path, without triggering any credentialed API call.
Alongside the new preview type, the existing preview plumbing was corrected: empty objects preview as empty instead of erroring, appended logs re-preview at their new length instead of the stale listed size, and metadata races when flipping between objects are gone. The preview and share dialogs now receive the object’s real size (92e8f4e65).
Downloads That Cannot Lie
The whole download path — progress reporting, single objects, folders, archives, byte ranges — was rebuilt around one principle: a download either completes correctly or fails visibly.
Honest progress
A missing, invalid, or contradictory total no longer becomes a fabricated percentage: the progress indicator stays indeterminate until the real total is known. Zero-byte objects are normalized deliberately instead of falling into the unknown-total path, and abort/cancel are terminal states — a late progress event cannot resurrect a cancelled download. Download requests settle exactly once, JSON error blobs are parsed safely, and generated object URLs are revoked.
Streaming folder downloads
Single-folder downloads now use the browser’s native streaming download path instead of accumulating the entire ZIP in JavaScript memory — a multi-GB folder no longer risks tab death. One consequence of the handoff model: the console’s transfer manager reports the folder download complete once it hands the stream to the browser (a toast says so), and the browser’s own download UI takes over from there — cancelling in the console after handoff does not stop the browser-side transfer. Multi-selection downloads still ride the existing POST response and therefore remain an in-memory Blob; changing that requires a separate API decision and is out of scope here.
ZIP integrity — a deliberate behavior change
This is the one change most likely to be noticed as “downloads broke”. In 2.1.1 and earlier, a folder/ZIP download that failed to read some objects silently skipped them and delivered an HTTP 200 archive missing files. In 2.2.0 any per-object failure — list, stat, read, entry creation, close, or copy — aborts the archive: a clean error if nothing was sent yet, an aborted connection mid-stream, so a truncated ZIP can never pass for a complete one.
The practical consequence: a user whose policy grants List on a prefix but GetObject on only a subset of it — a common IAM setup — previously received a partial archive; now the download fails at the first denied object. That was silent data omission, and 2.2.0 treats it as the bug. Download exactly what you can read, or scope the folder download to a readable prefix.
Byte ranges and status codes
- Malformed or unsatisfiable non-empty
Rangeheaders now return416withContent-Range: bytes */Ninstead of a 500. Range parsing is stricter than Go’s lenient default: signs, embedded whitespace,bytes=-0, and empty range elements are rejected. - A range request against a zero-byte object returns an empty 200 instead of a 500 — this was the empty-object preview bug.
- Failures from the lazy object
Statnow surface the real S3 status (403, 404, …) instead of a blanket 500. 206responses now setContent-Lengthbefore the header flush, so partial responses carry correct framing.- Object sizes are now always serialized — REST and WebSocket listings report
"size": 0for zero-byte objects instead of omitting the field, and the UI displays0 B. This is the ground truth the honest-progress work stands on.
Version history
S3’s valid null version ID stays visible instead of being dropped, version counting filters prefix matches to the exact object, and history is retained after bucket versioning is suspended or disabled.
Permission Gates That Actually Gate
Row actions were never disabled
Every screen with a data table passed its permission predicate through a prop name (disableButtonFunction) that the table component no longer reads — so view/edit/delete buttons rendered enabled regardless of permission, on every one of the 10 affected screens (Users, Groups, Policies, IDP, webhook settings, bucket access/replication/lifecycle panels). The predicates are now wired to the prop the component actually honors, and a source-guard test fails the build if the dead prop name ever reappears. Server-side authorization was never affected — the buttons produced errors when clicked — but the UI now communicates permissions instead of lying about them.
Independent service-account capabilities
Access-key management previously keyed several UI decisions off one combined permission check. 2.2.0 derives four independent capabilities — List, Create, Update, Remove — in one module and applies them consistently across the self-service Account screen and the admin user-details screen:
- the service-accounts tab shows when you can list, the create button when you can create, row selection and bulk delete only when you can remove, and the edit pencil only when you can update;
- View is now genuinely read-only: all fields disabled, no submit path, Enter is inert — previously “view” opened the editable dialog;
- the session’s advertised Create Access Key capability is computed correctly for request-scoped policies: a
Denyconditioned onsvc:DurationSeconds(an expiry restriction that can only be evaluated once a concrete request exists) keeps the capability visible — including with wildcard admin actions — while unconditional or login-time denies now properly hide it. The server remains the final authority at request time. The previous code kept the capability visible for any conditional deny, which over-advertised it. - an OIDC create/list/get/delete integration regression now runs through the UI’s explicit-credential endpoint, including the expected self-update denial.
Validate IAM policies before writes
Named-policy and service-account policy writes now reject malformed documents and bare S3 resource ARNs with a client error before making an Admin API request. Historical policy reads stay permissive for compatibility, but an incompatible stored policy must be corrected before it can be saved again. The strict parser and resource checks live in the console’s write path instead of importing fork-only policy APIs, preserving the advertised upstream minio/pkg v3.6.1 source-build floor.
Anonymous pages behave anonymously
Anonymous object-browser pages no longer issue protected Object Lock/retention requests that could only produce Access Denied noise, and they gained the language (文/A) and dark-mode controls.
User Status and Groups Are Separate Operations
Toggling a user’s enabled/disabled status and editing their group membership were one combined PUT /user/{name} call that required both payloads and — through the combined permission gate — demanded admin:EnableUser merely to edit groups. 2.2.0 splits them:
PUT /user/{name}/status(new) changes only the status, validates the value againstenabled|disabled, and rejects the signed-in user’s attempt to enable/disable themselves. Failed toggles no longer leave the switch out of sync — the UI reflects the server’s answer, not an optimistic flip.PUT /user/{name}/groups(existing) is now the only thing group editing calls, and no longer requiresadmin:EnableUser.PUT /user/{name}survives unchanged on the wire as a deprecated compatibility endpoint for existing API consumers; an unknown status there is now a 400 instead of a 500.
See Compatibility for the API-contract details, including a caveat for regenerated Swagger clients.
Database Notification Forms That Emit What They Mean
The PostgreSQL and MySQL event-destination forms were rewritten around a shared DSN parser/serializer:
- Structured fields and the raw connection string are one state: the raw string stays authoritative until a structured field is edited, at which point a canonical DSN is rebuilt — libpq keyword/value quoting for PostgreSQL, IPv6-bracket-aware
go-sql-driverformat for MySQL. Mode switches no longer mangle manually entered strings. - Generated previews mask credentials; the mask can never reach the API payload (the payload always uses the raw connection value).
- Saving requires both a connection string and a table, and cleared values actually propagate.
- The server now rejects DSNs its own configuration grammar would corrupt — embedded newlines, values that would parse as sibling config keys (
table=…inside a password), unbalanced quoting — with a 400 before anything is stored, and without echoing the submitted secret back. - Generic password and token fields across notification targets (Kafka, Redis, MQTT, NATS, webhooks) render as password inputs, including environment-overridden values.
Restart honesty
Configuration add, update, delete, and reset now honor the server’s actual restart-required answer instead of hardcoding “restart needed” (or worse, dropping it). The pending-restart flag is monotonic: once any operation requires a restart, later operations that don’t cannot clear it — only an actual service restart does.
Upload advisory
Browser uploads larger than 5 GiB get a non-blocking warning: the console uploads as one non-resumable request, and mcli with multipart upload is the right tool at that size.
Sessions, Metrics Auth, and i18n Hardening
- Sessions fail fast. An anonymous/empty session gets an immediate 401 instead of hanging on an empty-credential admin request. The console accepts the canonical
401and the legacy403invalid-session responses, and expired-session redirects are subpath-aware — a console served under/console/redirects within its base path. - Prometheus Basic auth works everywhere. The health check and the root-fallback probe now send Basic credentials (previously Bearer-only, so a Basic-auth Prometheus disabled every dashboard widget), Bearer tokens keep precedence, and every response body is drained so keep-alive connections are actually reused.
- Placeholder substitution is escape-proof by construction. Translated placeholder filling uses a one-pass literal formatter: values containing
$&,$1,$`, backticks, or braces stay literal, repeated placeholders all fill, and an AST source-guard test bans the unsafeString.replacepattern from ever returning.
Toolchain and Dependencies
Go 1.27 baseline
| Component | 2.1.1 | 2.2.0 |
|---|---|---|
go directive, build image, CI matrix |
1.26.5 / 1.26.x |
1.27.0 / 1.27.x |
golang.org/x/crypto |
v0.54.0 | v0.55.0 |
golang.org/x/net |
v0.57.0 | v0.58.0 |
golang.org/x/text |
v0.40.0 | v0.41.0 |
golang.org/x/mod |
v0.37.0 | v0.40.0 (closes CVE-2026-56864/-56865) |
golang.org/x/tools |
v0.47.0 | v0.49.0 |
The SILO forks
silo-pkg3.12.1 is the release dependency.pgsty/silo-pkgv3.12.1 was published on 2026-08-25 and the console pins it directly. It builds on the 3.12.0 release, which carries the policy resource-boundary hardening, condition-key lookup repair, LDAP and certificate-watcher fixes, and the Go 1.27 / etcd 3.7 baseline.- The
mclibrary moved to thepgsty/mcfork at a date-tagged pseudo-version. Import paths are unchanged. - The
requireline stays on upstreamv3.6.1deliberately. Go ignoresreplacedirectives in dependency modules, so a downstream module that requires this console resolves upstreamminio/pkg— and the SILOv3.12.1tag does not exist upstream. Requiring a real upstream tag keeps the console resolvable downstream; the replace applies the fork for the console’s own builds. The final CI verifies the public source-build surface against upstream v3.6.1. This is compile compatibility: a downstream build without its own top-level replace gets upstream behavior, not the fork’s SILO-specific IAM semantics. go-systemdis pinned back to v22.6.0: v22.7.0 usesCLOCK_MONOTONICon NetBSD, which doesn’t compile there; the pin holds until upstream ships the fix.
etcd 3.7.1 — client libraries only
All three etcd Go modules move together from 3.6.8 to 3.7.1, closing the TLS-listener denial of service GO-2026-6107 / CVE-2026-73500. etcd 3.7 removes legacy protobuf remnants and makes clientv3.New non-blocking — migrations that matter to consumers that construct clients or embed servers. The console does neither: its only etcd path is silo-pkg/quick operating on an already-created client with plain v3 Get/Put. This upgrades compiled client libraries only — it does not touch an operator’s etcd servers, cluster data, or deployment topology; a server upgrade to 3.7 still follows etcd’s own one-minor-at-a-time procedure.
Third-party maintenance stays conservative
The final release updates minio-go to v7.3.0 after a separate compatibility review, migrates its INI import path, and adds lifecycle-filter XML coverage. A console-side compatibility decoder accepts legacy AccountInfo tag payloads returned by older servers. Other accepted maintenance moves include jwx v2→v3 with httprc v3, go-openapi/swag/conv+typeutils 0.28.0, grpc-gateway 2.29.0, cheggaaa/pb 1.0.30, and go.yaml.in/yaml/v3 3.0.5. Larger unrelated go-openapi, pb/v3, compression, and test-library updates remain deferred.
On the frontend, the vulnerability workflow now covers pushes, manual runs, and development dependencies with immutable installs; vulnerable transitive resolutions were refreshed (fast-xml-parser 5.11, nanoid 3.3.18, @babel/core 7.29.7), dead exports and the unused http-status-codes dependency were removed. The new fast-xml-parser 5.x transitive tree (@nodable/entities, is-unsafe, anynum, fast-xml-builder, path-expression-matcher, xml-naming) was supply-chain-checked during this review: all six packages are published by the fast-xml-parser author’s own account and organization, their installed code is free of execution/network/exfiltration patterns, and the whole tree is development-only — nothing ships in the browser bundle.
Security Review
govulncheckat the release tree: zero vulnerable symbols reached, zero vulnerable imported packages. The Swagger build tool scans clean separately.- Closed by dependency moves: CVE-2026-73500 (etcd TLS listener DoS), CVE-2026-56864 / CVE-2026-56865 (x/mod verification).
- Still reported, still unreachable: the module-level GO-2026-5932 openpgp advisory — the console does not import
x/crypto/openpgp, and no fixed release exists. - The text preview was reviewed as an XSS surface (see above); its regression suite includes active-payload tests.
- The permission-gating fixes are UI-truthfulness fixes: server-side authorization was never bypassed in 2.1.1; the console simply displayed controls it shouldn’t have.
Compatibility
Nothing changes for deployment plumbing: no environment variable, configuration format, command, binary name, systemd unit, port, or embedded data layout changes. The release binary remains self-contained; Go 1.27.0 is a build-time requirement only.
HTTP API contract (console’s own REST API):
| Route | Change |
|---|---|
PUT /user/{name} |
Unchanged on the wire; now deprecated. OperationId renamed UpdateUserInfo → UpdateUserInfoLegacy, body model renamed to legacyUpdateUser (identical schema). Unknown status: 500 → 400. |
PUT /user/{name}/status |
New. Status-only body (enabled/disabled enum, 422 on violation), returns a user object populated with access key and status only — clients must not read group data from it. |
updateUser model |
Now status-only with enum — breaking for generated-spec consumers; the old shape lives on as legacyUpdateUser. |
| Object download | Error semantics changed: bad ranges 500→416 (+Content-Range: bytes */N), range-on-empty 500→200, Stat failures 500→real S3 status, ZIP failures silent-partial-200→visible failure, 206 responses carry Content-Length. |
| Listings & WebSocket | size always serialized, including 0. Additive. |
PUT /configs |
New 400 class for database DSNs the server’s config grammar would corrupt. |
| Config reset/delete | restart in the response now reflects the server’s real answer instead of always true. |
GET /session |
401 for empty-credential principals; the advertised Create-Access-Key capability is computed more strictly (see service accounts). |
If you generate client SDKs from swagger.yml: the UpdateUserInfo operation now points at /user/{name}/status with a status-only body. Code calling the generated UpdateUserInfo symbol keeps compiling but targets the new route; the old combined call is UpdateUserInfoLegacy. Audit call sites when you regenerate.
Behavior changes an operator may notice:
- Folder/ZIP downloads over partially readable prefixes fail instead of silently omitting unreadable objects (details).
- Range parsing is stricter than Go’s lenient default; degenerate range headers (
bytes=-0, empty elements) now get 416 instead of best-effort handling. - Database notification configs that only worked by accident (DSNs the config grammar mangled on the way in) are now rejected up front with a 400.
- The Create-Access-Key control is hidden for sessions whose policy unconditionally denies it (previously any conditional deny kept it visible).
- A pending restart-required indicator persists until an actual restart, instead of being clearable by a later unrelated config change.
- If a reverse proxy in front of the console compresses
/api/v1/…/downloadresponses, the text preview’s strictContent-Lengthverification will reject every preview as an error. The console itself only compresses static assets — leave API responses uncompressed at the proxy.
Regression Review
Because this release rewrites the download path and re-platforms the shared policy library, the full diff against v2.1.1 was re-reviewed adversarially in five parallel passes (Go API; object browser preview/download; permission gating and service accounts; forms/i18n/session; dependencies/build/CI), each hunting specifically for behavior that worked in 2.1.1 and silently changed.
Verdict: no unintentional regressions found. Every confirmed behavioral difference is one of the deliberate changes documented above. The review did surface two small pre-existing UI-guard defects, now visible because the dead disable-prop was brought back to life with predicates whose argument type was never right:
- the “cannot delete the Default IDP configuration” row guard compares the row object against the string
"Default"and therefore never engages (IDPConfigurations.tsx); - the “cannot delete an env-override webhook endpoint” row guard has the same object-vs-string mismatch and is inert (
WebhookSettings.tsx).
Neither is a 2.2.0 regression — both predicates were entirely dead in 2.1.1 — and in both cases the server still enforces the real rules. They remain documented follow-up items after the release. Two sharp edges are recorded as known limitations rather than defects: the new PostgreSQL DSN parser accepts only canonical key=value syntax when populating structured fields (an unusual-but-libpq-valid DSN shows empty structured fields, and editing a structured field then rebuilds the DSN from those fields), and TestCafe/Playwright coverage asserts UI gating while live-server deny-path coverage remains the integration suites’ job.
Verification
The release decision combines local release-preparation evidence with remote gates run against the exact tagged tree, 7dc4258a6:
Local release-preparation gates — go build ./..., go vet ./... (plus -tags testrunmain), gofmt, golangci-lint, go test -race ./..., go tool swagger validate, govulncheck, TypeScript tsc, Playwright, Prettier, knip, release-tag cross-compiles for linux/amd64 and linux/arm64, and go mod verify.
Embedded-assets determinism — the frontend was rebuilt from source through the full pipeline (yarn build + embed optimization) and the result compared against the committed web-app/build: byte-identical, zero dirty files. The nine commits after the earlier 19047161f candidate changed Go compatibility, workflows, and browser-test timing, but not product frontend source or embedded assets.
CI, exact final tree — Workflow run 32888892876 reported 32 successful jobs and one explicitly disabled React-test placeholder. It covered lint, semgrep, Go and API tests, five cross-compile targets, Swagger drift, latest-MinIO source builds, distributed integration, site replication, hermetic SSO, the complete TestCafe permissions matrix, subpath-nginx, Playwright, and coverage. Vulnerability Check 32888899120 passed both jobs on the same commit.
Release pipeline — goreleaser run 32916237254 passed both jobs against v2.2.0 and published the public GitHub release.
Downstream contract — in a scratch tree with every replace removed, go mod tidy + go build ./... succeed against upstream minio/pkg v3.6.1, proving the fork replacement never leaks fork-only symbols into the public module surface.
Test-infrastructure repairs shipped in this cycle (so the gates above actually gate): the Docker-backed integration/replication/SSO suites are back behind the testrunmain build tag (a bare go test ./... no longer tries to start containers); the SSO gate is hermetic — no sudo /etc/hosts edits, no ad-hoc pip installs, pinned SILO image, own port, real teardown; the integration gate asserts the new 416 range semantics and stopped binding its PostgreSQL fixture to a host port; the browser gates run off-Linux by publishing fixture ports; a stale pre-fork “MinIO administrator” selector was fixed; state-mutating TestCafe suites are serialized; Playwright CI installs the committed lockfile immutably.
Release Artifacts
The v2.2.0 GitHub release publishes:
- six standalone binaries: Linux amd64/arm64/armv6, macOS amd64/arm64, and Windows amd64;
- nine Linux packages: DEB, RPM, and APK for amd64, arm64, and armv6;
silo-console_2.2.0_checksums.txt, plus the SHA-256 digest GitHub records for every asset.
Those assets, the tag, and the release page are verified here. This page does not claim a separately distributed container image or detached signatures.
Related Commits
16960f7ab— fix: keep unknown downloads indeterminate5968bb37d— chore(deps): align the SILO Go dependency stack288ab1240— fix: harden sessions, metrics, and translationsecf3bb492— fix: harden object previews and downloads902d9650d— chore: tighten dependency and test gates194c70c7a— build: prepare SILO Console v2.2.0927b44e26— fix: harden database notification formsda2191be9— fix: clarify console upload and secret limits6141c2445— build: refresh SILO Console v2.2.0 assets097e76155— chore(deps): bump the shared package fork to v3.12.099ca523d6— fix: split user status updates out of the combined user routef4097992f— fix: honor the server restart result for configuration changesf1280032a— fix: restore permission-gated table row actions24ce0af97— fix: keep request-scoped access key conditions visible in the console92e8f4e65— fix: pass the object size into the preview and share dialogs8f6fb3c78— test: make the SSO gate hermetic and pin it to a SILO release384a2cb95— build: refresh SILO Console assets and record the changesa73cda376— test: fix the integration gate’s stale range and host portcf5049c1d— test: make the browser gates runnable and fix a stale selector6fa19d857— fix: complete service account permission boundaries7e57771a4— build: refresh SILO Console assets57cfe7aa0— fix: restore downstream and browser release gates19047161f— test: stabilize permissions browser gatese37dec873— fix: validate IAM policies before writes28505ed23— chore: update minio-go to v7.3.016abb971e— ci: harden validation and release gates2ddfcd036— fix: accept legacy AccountInfo tag payloads31332bca9— fix: preserve policy source compatibility3a8251086— ci: allow permission tests to finishc159fff78— ci: serialize shared-role permission tests2e91cdf9a— test: wait for watch controls to become ready7dc4258a6— test: allow asynchronous UI controls to settle
Links:
2.8 - Silo Console 2.1.0 Released
Published: 2026-08-06 · Version: v2.1.0 · Repository: pgsty/silo-console
SILO Console 2.1.0 is the first feature release after the independent 2.0.0. It does three things:
- Speaks two languages — every console screen, help topic, and documentation link now renders in English or Chinese, behind a toggle on every page, with zero new runtime dependencies;
- Reads the right metrics — the dashboard moves off the MinIO Metrics V2 names onto V3, with explicit handling for the semantics V3 changed underneath it;
- Stops lying in edge cases — a select-all that matched what a bulk action would delete, placeholders that survive object names containing
$&, timestamps that carry a timezone, and empty metrics that read “no data” instead of a fabricated0.
This is a minor release. No environment variable, module path, API contract, binary name, or data layout changes. Upgrading is a binary or image swap.
A 2.1.1 patch follows this release
v2.1.1, published the same day, completes the legend hardening described below: a label placeholder the legend builder cannot resolve is now removed instead of leaking literal braces into the Traffic chart legends, the one remaining substitution branch is escape-proofed against label values containing $& or $1, and the License page reports the actual release version instead of 2.0.0. Nothing else changes — upgrade straight to 2.1.1, and everything in this note applies unchanged.
Rebuild your embedded assets if you vendor this console
2.1.0 fixes a packaging defect present on the main branch after 2.0.0: the go:embed payload still carried the 2.0.0 frontend build, so a binary built from an intermediate commit would serve the old UI. The released 2.1.0 artifacts are built from the regenerated payload and are unaffected.
A Bilingual Console
The console is an administration surface for an object store, and a large share of its operators read Chinese first. 2.1.0 makes the interface bilingual without importing an i18n framework — the embedded delivery model means every kilobyte is paid for in the binary. This is issue #6, which proposed i18next; the dependency-free substitution is the one deliberate deviation from it.
How it works
The design constraint was: no new dependency, no build step, no extraction pipeline, and partial coverage must never break the page.
- English source strings are the dictionary keys.
t("Create Bucket")looks up the Chinese entry; a missing key returns the English string unchanged. Coverage can therefore grow incrementally, and a typo degrades to English rather than to a raw key likeconsole.bucket.create. - Three dictionaries, one merge.
zh.ts(165 chrome entries),zhHelp.ts(247 help-topic entries), andzhScreens.ts(1,373 screen entries) merge with chrome taking precedence — about 1,785 entries in total. - The language preference mirrors dark mode:
localStorage→systemSlice→setLanguage. There is no browser-locale detection; the default is English, and the choice is explicit. - Central interception points rather than per-callsite edits: the page-header wrapper, confirm dialogs, help items, route definitions, and the dashboard’s panel renderer each translate on the way out. This is why 220 screen files could be localized without touching their business logic.
- Module split matters.
i18n/lang.tsholds pure primitives (translate,localizeUrl) and imports no store —systemSlicedepends on it, so importing the store back would form a cycle. The hooks (useT,useLanguage,useLocalizedLink) andinterpolate()live ini18n/index.tsx.
The toggle is a stroke-drawn 文/A icon mounted in the page header on every page and reused on the login page.
What it covers
Login and SSO flows, navigation and the command palette, the dashboard and every metrics panel, buckets and the full object browser (uploads, previews, sharing, versioning, rewind), users/groups/policies/access keys, configuration and event destinations, IDP and KMS, logs, health reports, speedtest, profiling, inspect, trace, watch, and the license page.
Beyond visible strings:
- Documentation links localize.
silo.pgsty.comlinks gain a/zhprefix in Chinese; the Pigsty site swaps domains (pigsty.io↔pigsty.cc). GitHub, MinIO, AWS, and YouTube links are left alone. - The help blog feed is per-language, fetching
/zh/blog/index.xmlin Chinese, with an independent cache per language. - The command palette stays searchable in both languages. Menu entries translate for display but keep their English originals as keywords, so “桶” and “buckets” both match.
- Chart legends translate only their static prefix.
translateLegendpreserves instance suffixes like[server:drive], and the data layer keeps raw legends so components that match on them for arithmetic (capacity summing) keep working. - Timestamps are unified, not merely translated — see below.
What it costs
Roughly +61 KB on the embedded payload (2.79 MB → 2.85 MB, +2.2%), zero new dependencies, and the dictionaries land in their own lazily-loaded chunk. The English rendering path is byte-stable: with the default language, output is identical to 2.0.0.
What stays English
Backend error strings (182 of them) are produced by the Go server and are not translatable from the frontend. A handful of strings hardcoded inside the vendored mds component library — the collapsed-menu “Sign Out” tooltip, and the data table’s “Columns”, “Loading…”, and ON/OFF toggles — remain English; two of them (“Sign Out”, “Actions:”) are swapped via a scoped CSS rule, but the rest would require patching the vendor.
Metrics V3 Migration
The dashboard queried MinIO Metrics V2 names. SILO deployments scrape V3 (/minio/metrics/v3), so the dashboard depended on an endpoint the monitoring pipeline no longer collected. 2.1.0 rewrites all 26 widgets onto the V3 catalog — 31 queries over 29 distinct metric names — and drops three widgets (51/61/62) that no layout ever referenced. This is issue #7; the Info-page half is #8.
The decision is V3-only: no runtime fallback, no probing, no version-selection knob. SILO Console targets SILO deployments, where the server, the scrape pipeline, and the console ship together. The SILO server keeps serving V2 endpoints for external consumers; the console simply stopped using them. A fallback would have been actively harmful — a metrics store retaining 15 days of V2 series would let an or-fallback silently read stale data.
The semantics V3 changed
Three properties of V3 break a naive name-for-name rewrite, and each needed a deliberate answer:
- Cluster groups are exported identically by every node.
/cluster/*metrics carry no server label and are not leader-gated, so an N-node scrape yields N duplicate series. Queries aggregate withmax()/min()— neversum(), which would multiply cluster totals by the node count. - Zero values are not exported at all. Any metric whose value is ≤ 0 is skipped. Offline drive counts, healing-drive counts, and erasure-set health simply vanish rather than reporting
0, which a stat card renders as an empty panel. Every affected query carries a companion guard so the panel reads a real0. - There is no
minio_heal_*namespace. The V2 heal activity signal was in-memory anyway — it reset on restart and bumped on any scan. It is replaced by two cards with defensible semantics: Erasure Health (baselined on write quorum) and Usage Data Age (how stale the scanner’s usage snapshot is).
Zero-state semantics
An adversarial review of the migration produced eight findings, all fixed before release. They share one theme — the difference between zero, no data, and not yet scanned:
- Capacity free/used baselines on the always-present total, so a full cluster reads
0 freeinstead of vanishing. - Online Drives is guarded against the all-offline case, where the zero-skip would erase the panel exactly when it matters most.
- Bucket and object counts guard on the usage group’s own freshness gauge, so a cluster that has not completed its first scan reads no data rather than a fabricated
0. - Empty single-value results render as
—, not0. - An empty size distribution no longer fabricates seven zero-height bins.
- Fractional rates stay visible (
parseFloataxis domain, two-decimal CPU formatter) instead of collapsing to0. - Sub-second Usage Data Age clamps to “1 second” instead of rendering blank.
A regression suite (api/admin_info_metrics_test.go) now pins every widget query to the V3 catalog, asserts widget-ID uniqueness, and enforces the per-widget guard taxonomy: health and traffic widgets need a nodes-online companion, usage counts need the usage-group freshness companion, and capacity needs the total baseline. The full mapping is documented in docs/metrics-v3.md.
Also fixed
- Widget 17 queried
sent_bytestwice and widget 11 queriedsyscall_readtwice — both internode/syscall pairs were transposed into duplicates. - Label-less matrices (the result of
max()aggregation) serialize with nometricfield at all, which crashed the frontend’s label extraction and produced a0 Bcapacity donut and an empty usage-growth chart. Guarded. - An unused per-widget Prometheus label-values prefetch stalled every widget request by up to a second. Deleted.
- The dashboard’s usage cards, chart controls, and dense Traffic/Resources panels were rebuilt on one grammar and now reflow through tablet widths.
Two server-side bugs were identified during this work and are tracked upstream rather than worked around here: minio_cluster_usage_buckets_since_last_update_seconds emits nanoseconds (the objects variant is correct), and V3 bucket-level sent/received traffic are transposed.
Correctness Fixes
Placeholders that survive real object names
String.prototype.replace interprets $&, $', $`, and $1 in the replacement value as directives. S3 keys legally contain $. So an object named report$&.csv did not render as itself — it re-injected the matched placeholder text into the output and corrupted the message. All 37 dictionary placeholder substitutions now pass the value through a function replacement, where no such interpretation happens. This was a latent bug in the original English UI, not something i18n introduced; the i18n audit is simply what found it.
A select-all that means what it shows
The vendored data table renders a plain untranslatable “Select” header whenever onSelectAll is absent — which was the case on all seven selectable tables. Worse, the naive fix is wrong: a select-all that replaces the whole selection drops rows hidden by an active filter, so the header checkbox and a subsequent bulk action can target different sets. The implementation toggles only the currently visible rows and preserves filter-hidden selections, so the header state can no longer imply a different set than the action would touch.
Timestamps with a timezone
Bucket, object, version, rewind, and access-key timestamps rendered as a mix of verbose English forms and — in several places — a 12-hour clock without AM/PM, which is simply ambiguous. All of them now render as yyyy-MM-dd HH:mm[:ss] (ZZZZ) in both languages.
A translation runtime that survives live data
t() also receives runtime strings: user agents, RSS titles, object names. Two hardening changes followed:
- misses return unchanged, unconditionally — the implicit
@contextsuffix stripping is gone, because it silently mutated live data that happened to contain@; - dictionary lookups are guarded with
hasOwnProperty, so a hostile input naming an inheritedObject.prototypemember (constructor,toString) cannot leak a function into the UI.
Interaction and accessibility
- An expired session opening a deep link bounced through
/loginand back, accumulating a redirect chain instead of landing on the form once (#1). - Collapsed sidebar buttons carried no accessible name; screen readers announced them as unlabelled (#4). Access Key inputs now declare their autocomplete intent instead of letting password managers guess (#5).
- Mobile metrics and bucket panels scroll instead of clipping (#3).
- The speedtest control row wraps instead of overflowing its card, its duration accepts seconds or minutes, and its size defaults to MiB to match its own unit list.
- Sidebar bucket rows use a virtual row pitch matching the 44px item, so selected and hovered highlights no longer overlap.
- Unit chips render the selected unit’s label rather than its raw value.
No SUBNET, No Telemetry
Upstream removed Subnet, Registration, and Call Home; this fork inherited that state but still carried three traces. 2.1.0 removes them:
- the health websocket’s
subnetResponsefield never addressed a subnet — it is a sentinel meaning “the report was assembled” — and is nowreportStatus: "ok"; - two help topics claimed the health report “uploads automatically to SUBNET” and that inspect output is “transmitted to SILO SUBNET”. Neither was true. They now describe what happens: the report is generated on the deployment and downloaded by the browser;
- the unreferenced
CONSOLE_SUBNET_PROXYconstant is deleted.
For the record, 2.1.0’s outbound network posture is unchanged and remains: no analytics, no telemetry, no beacons, no external scripts or fonts. silo-console update is still disabled. The release catalog is contacted only if SILO_RELEASE_SERVICE_HOST (or RELEASE_SERVICE_HOST) is explicitly set — there is no default. The only automatic outbound request the browser makes is the help panel’s blog feed, and only after a user opens the Blog tab.
Upgrade Guide
There is nothing to migrate. No environment variable, module path, protocol field, systemd unit, binary name, or data layout changes between 2.0.0 and 2.1.0.
Two things are worth knowing:
- The dashboard now requires Metrics V3. If your Prometheus scrapes only the V2 endpoints, dashboard panels will read no-data. Point the scrape at
/minio/metrics/v3; Pigsty-managed deployments already do. - The language default is English, chosen per browser and stored in
localStorage. There is no server-side default and no browser-locale detection, so no existing deployment changes appearance on upgrade.
Verification Scope
Before tagging, the full change set was reviewed and the following gates were run against the final tree: go build, go vet, golangci-lint (0 issues), the Go unit suite across all packages, gofmt, TypeScript type checking, the frontend production build, Prettier across all sources, dictionary duplicate-key checks, and a debug-leftover scan of the complete diff.
The 29 intermediate commits were restructured into 20 logical ones by pure tree operations, and the rebuilt tip was verified byte-identical to the pre-rewrite tree. The embedded payload was rebuilt twice from a clean directory and confirmed byte-identical, which is the property the release pipeline’s zero-diff gate depends on. The pre-rewrite history is retained in a backup ref.
The Metrics V3 migration was additionally reviewed adversarially by an independent model, and all eight findings were fixed (see Zero-state semantics); its queries were validated against a live metrics store with real cluster data.
Known Limitations
- The SSO end-to-end suite requires an external OpenLDAP/Dex/MinIO topology and was not run in that environment this cycle; the OIDC code paths are covered by unit tests.
- Backend error strings and several vendored
mdscomponent strings remain English (see What stays English). - Chinese translation covers the console’s own surfaces; help-topic bodies are translated, but the documentation pages they link to follow the docs site’s own language coverage.
- Two server-side V3 metric bugs (nanosecond bucket-usage age, transposed bucket traffic) are tracked upstream and are not worked around in the console.
- Automatic self-update remains disabled; upgrades are explicit.
Issues Closed
2.1.0 closes every issue filed against 2.0.0. Each carries a comment on the tracker describing the fix, the commits, and the coverage added.
| Issue | Resolution |
|---|---|
#1 — unauthenticated deep routes recurse /login |
Absolute, base-path-aware login destination; deep-link and subpath test coverage |
| #2 — stale Uptime, malformed legends, cramped menus | Uptime derived from real server state, legends resolve on the V3 name label, 32 px chart controls, popup width floors |
| #3 — 390 px viewport clips content | Scrollable metrics tab strip; bucket table with a deliberate mobile column budget |
| #4 — unnamed collapsed sidebar buttons | Labels visually hidden rather than removed from the accessibility tree; named, keyboard-operable collapse toggle |
| #5 — Access Key fields lack autocomplete metadata | Field-level username / new-password tokens in a dedicated autofill section |
| #6 — English/Chinese localization | Hand-rolled bilingual layer, zero new dependencies, English-as-key fallback |
| #7 — migrate monitoring queries to Metrics V3 | V3-only; 26 widgets, 31 queries, 29 metric names, guard taxonomy, regression suite |
| #8 — replace N/A Info metrics | Erasure Health and Usage Data Age, sharing the advanced dashboard’s widget results |
Three acceptance criteria are recorded as unmet rather than quietly ticked: web-app has no unit-test runner, so the i18n test suite (#6) and the focused constructLabelNames test (#2) would require introducing test tooling first, and #6’s contributor documentation for adding translation keys is not yet written.
Related Commits and Links
The complete v2.1.0 change set consists of 20 logical commits. The v2.1.0 tag additionally carries three later documentation commits that rewrote the repository README; they change no shipped behavior.
8764f5d— fix(web): stop recursive login redirects437c56c— fix(ui): make the dashboard and bucket list usable on narrow screens85fc0c6— fix(a11y): name collapsed sidebar controls and credential fieldse3fed07— fix(metrics): rebuild dashboard cards, chart controls, and layoutfa11576— feat(login): polish controls and legal attribution9fc17c1— feat(i18n): add hand-rolled EN/ZH core, dictionaries, and language toggle622c02e— feat(i18n): localize login, navigation, and the help system6a03719— feat(i18n): localize dashboard and metrics screens14b1c2d— feat(i18n): localize bucket and object browser screens0298062— feat(i18n): localize identity, configuration, and event destinations41094f6— feat(i18n): localize observability, admin tools, and shared componentse964992— feat(metrics): migrate the dashboard to MinIO Metrics V30b2251f— fix(i18n): harden the translation runtime for live data and chart legends9b60148— fix(console): unify timestamps on a timezone-carrying standard formatbf110ae— fix(console): give selectable tables a visible-rows select-all5fc8f22— fix(i18n): escape-proof all placeholder substitutionsfef8fab— fix(console): polish speedtest, sidebar, and help chromec4911e8— chore(console): drop SUBNET remnants from health reporting1d631c4— docs: record the SILO Console v2.1.0 changelog912d847— build: regenerate optimized embedded web assets
Links:
2.9 - Silo Console 2.0.0 Released
Published: 2026-08-04 · Version: v2.0.0 · Repository: pgsty/silo-console
SILO Console 2.0.0 is the first major release of this object-storage administration console as an independent project. Continuing from the georgmangold/console v1.9.1 maintenance line, it accomplishes three things:
- An independent identity — product name, visual system, documentation entry points, source attribution, and the release pipeline all move into the SILO project, while the Go module path, environment variables, and other compatibility contracts are deliberately retained;
- A redesigned interface — the login page, theme system, dashboard, and console details are reworked under one design language, backed by a regenerated brand icon set;
- Hardened engineering — the embedded frontend payload shrinks from roughly 10MB to 3.5MB, known dependency vulnerabilities drop to zero, and a batch of inherited defects — including a real runtime data race — is fixed.
Before publication this release went through two independent review passes: a full code review with commit-history restructuring, followed by an adversarial re-verification (exhaustive asset validation, HTTP semantics probing, full routing regression, and smoke tests against the published artifacts themselves).
Read the compatibility boundary before upgrading
The major-version change in 2.0.0 is about public identity and delivery contracts, not the object data format or the S3 protocol. Installation scripts that reference the old repository, binary name, or container image must be updated; existing integrations that use CONSOLE_MINIO_SERVER, CONSOLE_MINIO_REGION, github.com/minio/console, or the MinIO-compatible Admin API must not be search-and-replaced.
Why 2.0.0
This console originated as MinIO Console and was carried forward by the Alevsk/console and georgmangold/console community maintenance lines. SILO Console continues from there, maintained by the Pigsty community as the browser-based administration interface for SILO.
The version jumps from v1.9.1 to v2.0.0 because these public contracts change together:
- the product is now uniformly SILO Console, with the primary repository at
pgsty/silo-console; - the release binary changes from
consoletosilo-console; - release assets, checksums, package metadata, CLI descriptions, and project links all switch to SILO;
- in-product identity, help entry points, copyright attribution, source offers, and trademark notices are re-established.
The migration strategy is “clear external identity, restrained internal compatibility”: operators must take notice, but the underlying compatibility interfaces are not mechanically renamed.
Naming and Delivery Contracts
| Scope | Previous name or location | 2.0.0 contract |
|---|---|---|
| Product | Console / legacy MinIO Console | SILO Console |
| Repository | georgmangold/console |
pgsty/silo-console |
| Release binary | console |
silo-console |
| Binary assets | console-<os>-<arch> |
silo-console-<os>-<arch> |
| Checksums | console_<version>_checksums.txt |
silo-console_<version>_checksums.txt |
| Website and docs | upstream / previous maintainer | silo.pgsty.com and silo.pgsty.com/docs/ |
For container deployments, use pgsty/silo-console on Docker Hub. See the Console v2.4.1 release notes for the current version and pull command.
CLI authorship, usage text, and project descriptions now identify Pigsty and SILO Console. DEB/RPM/APK vendor, maintainer, homepage, description, and license metadata are updated accordingly; the executable installs to /usr/local/bin/silo-console.
Deliberately Retained Compatibility Identifiers
The following names still contain minio or the old console, but they are interface, protocol, or installation compatibility layers — not leftover branding:
| Surface | State in 2.0.0 | Reason |
|---|---|---|
| Go module | github.com/minio/console retained |
changing it breaks every Go import |
| Server endpoint | CONSOLE_MINIO_SERVER retained |
widely used by existing deployments |
| Server region | CONSOLE_MINIO_REGION retained |
existing compatibility contract |
| Other configuration | existing CONSOLE_* variables remain valid |
avoids migration with no benefit |
| S3/Admin API names | MinIO-compatible fields and enums retained | they describe the actual protocol |
| Development build | make console still produces ./console |
keeps developer workflows working |
| Package systemd unit | minio-console.service retained |
avoids duplicate services on upgrade |
| systemd user and config | console-user and /etc/default/console |
avoids unnecessary account/config migration |
Upgrade scripts therefore must not run repository-wide minio → silo or console → silo-console replacements. Migrating these compatibility interfaces in the future will require aliases, deprecation windows, and an explicit dual-read strategy; 2.0.0 does none of that.
A Redesigned Interface
2.0.0 is not a logo swap — the interface was redesigned end to end.
Login page
The login page is rewritten from scratch. The left brand panel renders a slowly drifting sine-mesh animation generated purely on Canvas (zero external dependencies, honors prefers-reduced-motion, pauses in background tabs), states the project’s proposition — “Keep the S3 Interface / Own the Object Store” — and keeps the full MinIO trademark notice at the bottom. The right-hand form is functionally untouched, preserving every existing automation selector. The Chakra Petch typeface used by the SILO wordmark ships as a ~20KB locally bundled subset with no external requests.
A unified theme system
All console colors converge into one light/dark theme layer: neutral greys for text and borders, the brand steel blue for primary actions and selection, and a sidebar that uses the same night palette as the login panel in both modes. Controls and cards share consistent radii and transitions, inputs get a keyboard focus ring, and modals animate in (also honoring reduced motion). Server-provided customStyles keep full precedence.
Console polish
- Dashboard (Metrics): stat cards rebuilt under one grammar — muted labels, tabular numerals, aligned status dots; charts and info strips are theme-driven; the upstream absolute-positioning layout is gone.
- Unified empty states: placeholder text in Watch, Trace, bucket Events/Replication/Lifecycle, and every other data panel is now centered and de-emphasized instead of raw top-left text.
- Vertical tabs: detail-page tabs change from bordered grey blocks to a quiet pill list, eliminating the stray empty cell at the bottom of the rail.
- License page: a new VERSION section shows both the connected server’s release and the Console’s own version; accounts without
admin:ServerInfonever issue the request and the row stays hidden. The page also consolidates AGPLv3 licensing, the AGPL section-13 source offer, lineage, and trademark boundaries. - A batch of interaction fixes: the sidebar now collapses on initial load at mobile widths (previously it waited for a resize event); the bottom navigation no longer lags window-height changes; the bucket accordion highlight spans the full row; the dashboard no longer overflows horizontally on narrow screens; and the help panel is now truly lazy — the login page makes no external requests at all.
Brand icon set
The favicon, PWA, and Apple Touch icons still carried a previous-generation hand-drawn emblem. 2.0.0 re-rasterizes every size (ico 16+32, favicon 16/32/96, apple 180, manifest 192/512) from the official silo.svg vector emblem, with safe-area margins on home-screen sizes, and trims the Web App Manifest to the modern icon set, dropping the 2014-era legacy density entries. The icon payload drops from 473KB to 160KB, and the browser tab icon finally matches the in-product brand.
Smaller and Faster
Embedded delivery is this console’s core form factor — the frontend ships inside the binary via go:embed. 2.0.0 optimizes that path systematically:
- Embedded payload: ~9.6MB → 3.5MB. Text assets (JS/CSS/SVG/…) are precompressed at build time with deterministic gzip and embedded compressed-only; legacy WOFF fonts (~1.25MB that no supported browser ever downloads) and a set of entirely unreferenced orphan images are removed.
- First-load transfer: ~5.7MB → ~1.7MB. Static assets previously shipped uncompressed on the wire; they are now emitted directly with
Content-Encoding: gzipat zero runtime cost, with on-the-fly decompression for the rare client that does not accept gzip. - Correct HTTP semantics. Accept-Encoding is parsed with full RFC 9110 q-values (
gzip;q=0gets identity bytes), responses carryVary: Accept-Encoding, and non-GET/HEAD requests to static paths and the SPA entry receive 405 with anAllowheader. - Reproducible builds. Compression uses a pure-JS implementation (fflate) for byte-identical output across platforms, and the release pipeline enforces a hard gate: rebuilding the embedded assets in a clean environment must produce zero diff against the commit.
Release binaries (all frontend assets included, stripped) weigh roughly 35–40MB; for the downstream SILO server, embedding this console now costs about 3.5MB instead of about 10MB.
Security and Dependencies
Go: the build baseline moves to Go 1.26.5 and the golang.org/x family is fully refreshed. Every reachable vulnerability reported by govulncheck is resolved:
| Dependency | Fixed version | Advisories |
|---|---|---|
google.golang.org/grpc |
v1.82.1 | GO-2026-6061 |
github.com/prometheus/prometheus |
v0.311.3 | GO-2026-5710 / -5662 / -5381 / -5264 (incl. remote-read DoS) |
github.com/klauspost/compress |
v1.18.7 | GO-2026-5841 |
The single remaining advisory sits in golang.org/x/crypto, has no upstream fix yet, and is unreachable from this codebase; it is tracked as a known item.
Frontend: the full dependency-tree audit (production and tooling) is clean, covering the high-severity form-data CRLF injection and the DOMPurify and qs advisories; React Router is migrated to 7.18.2 (keeping the v6-compatible declarative API, with full routing regression). The only explicitly ignored advisory affects an unstable API this project does not use.
Runtime correctness: a real data race between HTTP log-target initialization and shutdown is fixed, along with shared-mock races in the test suite; supported Go packages pass -race across the board. As a side benefit, the go-m1cpu upgrade fixes the local go run cgo crash on recent macOS.
Update Checks and Default Network Behavior
This release keeps conservative defaults for upgrade tooling:
- automatic self-update in
silo-console updateis disabled — the command prints guidance and never downloads or replaces the binary; - the release catalog gains
SILO_RELEASE_SERVICE_HOST, with the previousRELEASE_SERVICE_HOSTas a compatibility fallback; with neither set, no remote release service is contacted; - the help panel’s blog content loads only when opened, and its links accept
https://silo.pgsty.comexclusively.
Automatic updates will be reconsidered once signed release assets and a tested rollback path are in place.
Release Artifacts and Platform Matrix
The release ships 16 assets:
| Type | Coverage |
|---|---|
| Standalone binary | Linux amd64/arm64/arm, macOS amd64/arm64, Windows amd64 |
| System packages | DEB / RPM / APK × amd64/arm64/armv6 |
| Checksums | silo-console_2.0.0_checksums.txt (SHA-256) |
The pipeline triggers on tag pushes, pins third-party Actions to commit SHAs, and enforces the clean-checkout and zero-diff asset-rebuild gates before GoReleaser runs.
Upgrade Guide
Standalone binary
When building from source, make console still produces ./console; install it under the release name before wiring it into a production service.
DEB/RPM/APK and systemd
Packages continue to install /etc/systemd/system/minio-console.service, whose unit starts /usr/local/bin/silo-console. EnvironmentFile=/etc/default/console, console-user, and existing CONSOLE_* variables are unchanged. This retention lets package upgrades keep acting on the existing service instead of creating a parallel one.
Configuration and integrations
- do not rename
CONSOLE_MINIO_SERVERorCONSOLE_MINIO_REGION; - do not touch
github.com/minio/consolein Go imports; - prefer
SILO_RELEASE_SERVICE_HOSTfor self-hosted release catalogs; - replace any reliance on
console updatewith explicit download, verification, and deployment; - update process-path-based monitoring to
/usr/local/bin/silo-console.
This release does not change the object data layout and requires no bucket or object migration.
Dual Review and Validation Scope
2.0.0 went through two independent review passes before publication. The first pass performed a full code review, fixed the defects described above, restructured 13 intermediate commits into 8 logical ones, and ran Go -race across supported packages, go vet, golangci-lint, govulncheck, frontend type checks, production builds, Prettier, dead-code checks, and the full dependency audit. The second, adversarial pass independently re-ran the core gates and added:
- all 184 embedded files fetched three ways each (gzip client, identity client, HEAD) with per-file hash comparison against the embedded sources;
- RFC semantics probes (including combined q-values such as
gzip;q=0, *;q=0.5), method restrictions, the OIDC callback, and SPA deep links; - full React Router 7 regression: deep links, client-side navigation, bucket-detail tab switching, and browser history back;
- mobile first-load sidebar behavior, login-page external-request monitoring, and light/dark full-site tours;
- downloaded release assets verified byte-for-byte against checksums, binary self-reported version confirmed, and a smoke test of the published binary against a live server;
- zero-diff asset rebuilds confirmed on both macOS and Linux.
The complete pre-rewrite history is preserved in backup refs for rollback.
Known Limitations
- automatic self-update is disabled; upgrades are explicit;
- the SSO end-to-end suite requires an external OpenLDAP/Dex/MinIO topology and was not run in that environment this cycle (the OIDC code paths are covered by unit tests and HTTP-level checks);
- one
golang.org/x/cryptoadvisory has no upstream fix yet and is unreachable from this codebase; - SILO does not yet maintain its own video library; videos in the help panel are clearly labeled upstream compatibility material;
- administrative features depend on the MinIO-compatible Admin API — SILO Console is not a generic browser for arbitrary S3 services;
- retained Go module paths, environment variables, protocol fields, and the systemd unit name still appear in code, configuration, and process listings.
Related Commits and Links
The complete v2.0.0 change set consists of 8 logical commits:
50797de— feat: establish SILO Console identity and compatibility23ae6e8— feat: redesign and harden the SILO Console web app7a83a77— build: update Go toolchain and dependencies1330d25— fix: eliminate logger shutdown and test mock races06b3a34— docs: publish the SILO Console v2.0.0 guide4b24372— build: regenerate optimized embedded web assetsc38eb64— ci: package and publish SILO Console v2 releasesb952a12— brand: regenerate the icon set from the official silo.svg emblem
Links:
2.10 - silo-pkg 3.13.0 Released
Follow-up, 2026-09-13: this article preserves the v3.13.0 release-time migration record. All four maintained components have since adopted the own-module path; current pkg is v3.14.0. The “not yet migrated” statements below describe that earlier date; see the current component matrix.
Release date: 2026-08-30 · Version: v3.13.0 · Commit: 215f116 · Repository: pgsty/silo-pkg
Version 3.13.0 is a breaking release with one theme: the module stops claiming upstream’s import path and takes its own. It also retires the pgsty/silo-go fork, which no longer carried anything upstream lacks.
This package release and a SILO server release are different gates. The package tag and GitHub Release are public. silo, silo-console, and mcli have not yet moved to the new import path; they continue to build against v3.12.2 through the old replace arrangement. No SILO server tag, container image, package set, deployment, or production rollout is established by this article.
Release at a Glance
- Published: the
silo-pkg v3.13.0tag and GitHub Release, declaringmodule github.com/pgsty/silo-pkg/v3. - Retired: the
github.com/minio/pkg/v3module identity and thereplacedirective every consumer had to repeat. - Retired: the
github.com/pgsty/silo-go/v7fork; this release requires upstreamminio-godirectly. - Unchanged: every package, symbol, and behaviour. Only the path they are imported from moved.
- Not part of this release: the matching
silo,silo-console, andmclimigrations.
Why the Path Moved
The fork kept upstream’s github.com/minio/pkg/v3 path so it stayed a drop-in replacement, selectable with one replace directive. Go does not inherit replace directives from dependency modules, and three costs came due while preparing the next SILO server release.
Every consumer had to repeat the redirect, and one that forgot built against upstream silently. A module that requires this package but omits the replacement resolves the real minio/pkg, compiles, and quietly loses the fork’s behaviour.
The require line had to name a version the source no longer matched. ParseConfigStrict first appears upstream in v3.11.0, and Resource.IsBareARN exists in no upstream version at all. A require github.com/minio/pkg/v3 v3.6.1 next to source that needs v3.12 APIs is metadata that cannot be made true by editing the number.
Working around that propagated. pgsty/mc carried a compile-time sentinel — a reference to Resource.IsBareARN — purely to turn the silent downgrade into a build failure. Raising mc’s floor to make its metadata honest pushed the requirement through the module graph and forced silo-console’s deliberately low floor upward, which is exactly what that floor existed to prevent.
Upstream’s path had little to offer in return here. minio/pkg is a small internal library, and this module’s consumers are silo, silo-console, and mcli. The Silo Go SDK keeps upstream’s github.com/minio/minio-go/v7 path, where drop-in compatibility is worth having and upstream is actively maintained.
Upstream minio-go Replaces the Silo Go SDK
pgsty/silo-go no longer carried any functional divergence. Its one unique change, Return CopyObject checksums in UploadInfo, was merged upstream on 2026-08-24. Everything else in the fork was a version string, a logo, a README, and a lint-tool block.
Upstream’s newest tag v7.3.0 predates that merge by 14 commits, among them a data race fix in parallel multipart checksum hashing, so pinning the tag would be a regression. This release therefore requires the pseudo-version v7.3.1-0.20260828014306-0e78d3f18efe and will move to a tag when upstream cuts one.
Compatibility
Every package, exported symbol, and behaviour is unchanged. This release moves where they are imported from, nothing else.
Take it together with the matching silo, silo-console, and mcli changes. A consumer that upgrades alone will not build, because the old and new paths are different modules and types do not cross between them.
Before:
After:
…and rewrite github.com/minio/pkg/v3/... imports to github.com/pgsty/silo-pkg/v3/.... Consumers that stay on the old arrangement keep building against v3.12.2 and earlier, which remain published.
One consequence is worth expecting. Because the two paths are now separate modules, a build can contain both — a third-party dependency that imports github.com/minio/pkg/v3 no longer has its import redirected. In the SILO stack this happens once, through minio/colorjson and minio/dperf, both of which reach only pkg/v3/console. That package’s colour switch is fatih/color’s process-wide NoColor, which every copy shares, so disabling colour still disables it everywhere.
Verification
go build ./...,go vet ./...,go mod tidy -diff, andgofmt -l .are clean;go test ./...passes 23 packages.- CI on the release commit: Go, LDAP Config Validator, VulnCheck.
- The new path was resolved from the module proxy by a fresh module with no
replacedirective, which importedpolicyand ran. - The downstream migration was validated end to end before this release:
mcli(194 files),silo-console(36 files), andsilo(181 files) all build, vet, and pass their suites.silo’s rebrand guard passes with a compatibility-baseline diff of exactly 19 deleted import entries and no change to environment variables, metrics, headers, routes, policy values, or exported symbols.
2.11 - silo-pkg 3.12.0 Released
Release date: 2026-08-24 · Version: v3.12.0 · Commit: 2b087a1 · Repository: pgsty/silo-pkg
Version 3.12.0 is a main-line minor release with two themes: a policy-write guard for ARN prefixes that name no resource, and the maintained Go 1.27 / etcd 3.7 dependency baseline. It adds two exported inspection/validation methods, raises the verified consumer floor to Go 1.26, and is the first silo-pkg release whose strict validation path is enabled by the SILO server for named-policy and service-account policy writes.
This package release and a SILO server release are different gates. The package tag and GitHub Release are public. SILO main consumes it in eee05a17c, with the operator note recorded as SN-2026-005. No date-style SILO server tag, container image, package set, deployment, or production rollout is established by this article.
Release at a Glance
- Published: the
silo-pkg v3.12.0tag, GitHub Release, and strict library validation for bare ARN prefixes. - On SILO remote
main: named-policy and service-account policy write integration. - Deliberately permissive: existing IAM policy loading, IAM import, and site-replication receive paths.
- Deferred: STS inline-policy strict validation and Console-side early validation; the server remains authoritative.
- Not part of this release: a SILO server binary, image, package, deployment, or production rollout.
Bare ARN Prefixes Were Policy No-ops
An S3 resource ARN needs a resource after its namespace prefix:
The policy parser also accepted the prefix by itself:
That string names no bucket or object. The existing parser normalized it to the wildcard resource type while retaining arn:aws:s3::: as the match pattern. Real S3 authorization candidates look like bucket or bucket/object, so the pattern normally matched nothing even though the policy validated successfully.
For statements that actually perform resource matching, the impact depends on Effect and on whether the prefix appears in Resource or NotResource:
| Statement shape | Existing runtime result |
|---|---|
Allow + bare Resource |
Grants nothing |
Deny + bare Resource |
The intended denial does not fire |
Allow + bare NotResource |
Excludes nothing and can grant far more than intended |
Deny + bare NotResource |
Can deny far more than intended |
The dangerous cases are policy-dependent fail-open behavior, not an unauthenticated remote exploit and not a CVE. A policy author, template, or automation must first submit the malformed resource. The same issue applies to registered S3 Tables and KMS ARN prefixes.
Resource-less admin actions, sts:* action statements, and the first phase of two-step KMS authorization bypass resource matching; a bare prefix does not change their existing runtime decision. Strict writes still reject the deceptive field so a policy cannot look scoped when that scope is ignored.
The Historical *arn:... Spelling
On the permissive compatibility path inherited from earlier releases, serializing a parsed bare prefix adds the wildcard type marker:
Re-parsing either spelling produces the same internal resource value. The 3.12 guard therefore recognizes both the exact prefix and its historical star-prefixed serialization. This matters for stored/exported policies and for clients that parse and marshal a document before sending it to the server.
The accepted wildcard corpus remains unchanged: *, **, ***, *foo, and explicit resources such as arn:aws:s3:::* still parse as before.
A Strict Write Path, Not a Storage Migration
The fix deliberately separates policy loading from policy creation:
ParseConfigandValidateremain permissive. Existing stored policies keep loading and evaluating with the same matching and serialization behavior.ParseConfigStrictandValidateStrictreject a registered ARN prefix that names no resource, in bothResourceandNotResource.Resource.IsBareARN()detects the normalized exact/historical form without changing the exportedResourcestructure,ParseResource, matching, or JSON representation.ResourceSet.ValidateStrict()exposes the strict resource-set check to consumers.
Keeping the existing resource representation is important for mixed-version sites: v3.11 and v3.12 nodes continue to compare and serialize stored policies the same way, so this fix does not create a site-replication mismatch or require a storage migration.
SILO Enables the Guard on Three Boundaries
SILO commit eee05a17c selects silo-pkg v3.12.0 and uses strict parsing when:
- creating or replacing a named IAM policy;
- creating a service account with an inline session policy; and
- updating a service account’s inline session policy.
Compatibility-sensitive paths stay permissive in this rollout:
- loading named policies and embedded policies already at rest;
- IAM import/restore;
- site-replication receive and apply paths;
- STS inline session policies; and
- bucket policies, whose existing bucket/action validation already rejects these forms.
Enabling ParseConfigStrict also activates two pre-existing admin-policy checks: one admin statement may not contain both Resource and NotResource, and a bucket-scoped admin action may not use a non-S3 resource. These are intentional authorization tightenings and are documented in the advisory ledger under SN-2026-005.
In this article, bare ARN prefix means an ARN namespace with no resource after it, such as arn:aws:s3:::. It is different from the valid bare bucket ARN used in the 3.11 bucket/object-boundary fix, such as arn:aws:s3:::my-bucket.
Operator Action
Existing policies are not rewritten automatically because the intended resource cannot be inferred. Before deploying a SILO server build that contains the strict integration:
- inspect named IAM policies for exact or historical bare prefixes;
- inspect service-account inline policies;
- replace each finding with the intended concrete resource; or use a suffix wildcard only if all resources in that namespace are truly intended; and
- repeat the audit after all sites have completed the rolling upgrade.
Do not blindly turn every finding into arn:aws:s3:::*: that could replace an inert statement with a cluster-wide grant or denial. A legacy policy containing a bare prefix still loads, but it cannot be submitted unchanged through the three strict write endpoints; correct it before editing another property on the same policy or service account.
The safer audit path uses policy-info and access-key-info APIs rather than a full IAM export, because a complete export contains user and service-account secrets. STS strict validation remains deferred until live machine clients and their session-policy templates can be audited separately.
Why This Is a Minor Release
This is v3.12.0, not v3.11.1, because the release combines three compatibility-relevant changes:
- the etcd client crosses from the 3.6 minor line to 3.7;
- the module’s verified
gofloor rises from 1.25.0 to 1.26.0; and - the package adds exported bare-ARN inspection and strict resource-set validation APIs.
The module path remains github.com/minio/pkg/v3; the /v3 import suffix and every existing import site stay unchanged.
Go and Dependency Baseline
The go and toolchain directives have separate purposes:
go 1.26.0is the supported consumer floor required by the selected etcd 3.7 modules.toolchain go1.27.0is the maintained development and CI baseline.- CI actions move to the Node 24 runtime.
Key selected versions change as follows:
| Module | 3.11.0 | 3.12.0 |
|---|---|---|
go.etcd.io/etcd/{api,client/pkg,client}/v3 |
3.6.6 |
3.7.1 |
golang.org/x/crypto |
0.54.0 |
0.55.0 |
golang.org/x/net |
0.57.0 |
0.58.0 |
golang.org/x/text |
0.40.0 |
0.41.0 |
github.com/minio/minio-go/v7 |
7.0.97 |
7.0.99 |
github.com/minio/mux |
1.8.2 |
1.9.2 |
github.com/cheggaaa/pb |
1.0.29 |
1.0.30 |
github.com/lestrrat-go/jwx/v3 |
3.0.12 |
3.0.13 |
github.com/lestrrat-go/httprc/v3 |
3.0.1 |
3.0.6 |
github.com/grpc-ecosystem/grpc-gateway/v2 |
2.27.3 |
2.29.0 |
go.uber.org/zap |
1.27.1 |
1.28.0 |
Smaller selected updates include uax29 2.3.1, fastjson 1.6.10, secp256k1 4.4.1, and goccy/go-json 0.10.6. The old lestrrat-go/option v1, gogo/protobuf, and stale test-only requirements leave the selected graph.
etcd 3.7.1 is the first fixed release on the 3.7 line for GO-2026-6107 / CVE-2026-73500, an unauthenticated TLS-listener denial of service. Updating these Go client modules does not upgrade an operator’s external etcd server. SILO does not use the removed grpc.WithBlock behavior, and client compatibility with an etcd 3.6.14 server was exercised during release validation.
The dependency graph declares coreos/go-systemd 22.7.0, but the module replaces it with 22.6.0 because 22.7.0 does not compile on NetBSD. The SILO server carries the same portability override.
Compatibility
- Import paths and the module major remain unchanged.
- Existing policies keep loading and evaluating unchanged; only strict create/update calls reject the malformed prefixes.
- No policy, IAM database, wire protocol, or etcd data migration is performed.
- Downstream users of
grpc.WithBlockin etcd client dial options must migrate to a supported readiness check; SILO does not use it. - A real external etcd cluster upgrade remains a separate operational procedure.
- Upstream AIStor Memory and new AIStor-only S3 Tables action vocabularies are not imported by this fork.
Consumers select the release with:
Verification
The tagged package passed:
- the repository’s complete
make testgate: pinned lint plusgo test -race -tags kqueue ./...; - targeted bare-ARN tests repeated to disturb Go map iteration order;
go mod verify,go vet,git diff --check, andgovulncheckwith zero reachable vulnerabilities; and- an implementation-level Claude Opus Max review with a GO verdict and no P0/P1 findings.
The SILO integration passed:
- the complete IAM server suite, including exact/historical named-policy rejection and service-account create/update rejection;
go test ./cmd -count=1,go vet ./cmd, andgo test ./...;- golangci-lint 2.13.1 with zero findings,
go mod verify, andmake check-gen; and - a second Claude Opus Max review with a GO verdict; mutation tests proved all three strict call sites and their integration assertions are load-bearing.
Direct VCS module resolution verified v3.12.0 at commit 2b087a1 with module checksum:
The release environment could not reach proxy.golang.org or sum.golang.org because those connections timed out, so public-proxy observation is not claimed as release evidence. The Git tag, GitHub Release, direct module archive, origin commit, and checksum were verified.
Related Changes
2bc3a91: move CI actions onto Node 24c8c6872: align the SILO Go dependency stack2b087a1: reject bare ARN prefixes on strict policy writes; taggedv3.12.030c49bd: update the README dependency example after the tageee05a17c: enable strict named-policy and service-account writes in SILO56c67dacf: recordSN-2026-005
What Is Not Released Here
This article does not claim a new SILO server version, binary, package, container image, deployment, production rollout, Console release, or strict STS rollout. Those remain separate release gates and must be reported separately when completed.
2.12 - Silo Pkg 3.11.0 Released
Release date: 2026-08-04 · Version: v3.11.0 · Commit: d8b1fa7 · Repository: pgsty/silo-pkg
This is the fork’s first pinned release. It restores the IAM bucket/object resource boundary reported as upstream minio/minio#20449: a policy condition-key bypass fix, three LDAP connection defects, a certificate watcher leak, a seeded-RNG defect, and the module’s real minimum Go version.
Two things to check before upgrading
- This release tightens authorization. Twelve bucket-level write actions are no longer reachable through an object-only resource pattern such as
arn:aws:s3:::bucket/*. If you write your own bucket-scoped policies, read The IAM bucket/object boundary — the fix is one line of policy for anyone affected, andMINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=onrestores the previous behaviour in full. - The condition-key fix still needs its server half. The policy lookup change and the server changes that reserve internal condition-key names each cover one half of that problem. The companion server work exists in
pgsty/miniocommit2f55347f7but is not yet on publicorigin/master, and no published Silo server release contains it. Verify that a later server release explicitly includes it.
What This Repository Is
silo-pkg is a maintained fork of minio/pkg, carrying fixes needed by community MinIO forks that the now commercially driven upstream no longer accepts. The repository was renamed from pgsty/minio-pkg on 2026-08-02.
The module path intentionally remains unchanged as github.com/minio/pkg/v3. Existing import "github.com/minio/pkg/v3/..." statements do not change; only the right-hand side of the replace directive does:
The /v3 suffix is the module’s major version, not a directory name, and must not be omitted. It is also why this release is numbered v3.11.0 rather than v4.0.0: Go requires the major version of a tag to match the major-version suffix declared in go.mod, so a v4.0.0 tag on a .../v3 module is rejected by the toolchain. Publishing a real v4 would mean changing the module path and rewriting roughly 395 import sites across the server, mc and Console — abandoning the drop-in property that is the point of keeping upstream’s path.
The IAM Bucket/Object Boundary
Every bucket-level S3 operation authorizes with an empty object name. The IAM matcher turned that into a resource string and, for the empty-object case, appended a trailing slash:
"bucket/" is matched by the wildcard pattern "bucket/*", because * matches the empty string. A policy granting s3:* on arn:aws:s3:::bucket/* — which reads as “anything, but only on the objects in this bucket” — therefore also authorized bucket-level actions. In a multi-tenant cluster, a tenant holding only that grant could call PutBucketPolicy and install {"Principal":"*"}, making the bucket publicly readable or writable, or grant itself bucket-level control. It could also delete the bucket outright, which is the reproduction in the upstream issue.
The bucket-policy evaluation path used for anonymous access never had this slash and was already reference-correct. Only the IAM path was wrong, in exactly one place.
Why not correct the whole boundary
Removing the slash for every bucket-level request is the obvious fix, and upstream tried it: the change was reverted the same day for breaking policies that relied on the old behaviour. Two properties make the full correction a migration rather than a patch.
It revokes grants real deployments depend on. It does not only revoke the dangerous bucket writes — it also revokes ListBucket, GetBucketLocation and ListBucketMultipartUploads when granted through bucket/*. The evidence is upstream’s own test suite: eleven STS integration tests grant s3:ListBucket on bucket/* and then assert that listing works. If the project that wrote the server writes it that way, production policies do too.
It cuts both directions. The matcher builds the same resource string for Allow and Deny, so removing the slash tightens over-granting Allow statements and simultaneously loosens over-blocking Deny statements. An administrator who locked a bucket with Deny s3:* on bucket/* would silently lose that protection.
How the protected set was chosen
The scope was decided by one question: does reaching this action give the caller something its object-scoped grant does not already provide?
That question is the right one because of how the defect fires. Resource matching runs after action matching, so the bug only bites when the statement already grants the bucket-level action — which in practice means s3:*. The affected principal therefore already holds full read, write and delete over every object in the bucket. The useful question is not how dangerous an action sounds in the abstract, but what reaching it adds to a position that already includes all of the data.
Withheld from object-only grants (twelve actions):
| Action | Why it qualifies |
|---|---|
PutBucketPolicy, DeleteBucketPolicy |
Hand access to other principals, anonymous included, and can grant the caller bucket-level actions it was never given. Self-escalation and public exposure. |
PutBucketObjectLockConfiguration, PutBucketVersioning |
Defeat protections that exist precisely to stop a holder of write access from destroying data. |
PutReplicationConfiguration, PutLifecycleConfiguration |
Act under server credentials and keep acting after the caller’s access is revoked. |
DeleteBucket, ForceDeleteBucket |
Destroy the bucket entity and its configuration irreversibly. The reproduction in the upstream issue. |
PutBucketCors, DeleteBucketCors, PutBucketQOS, PutInventoryConfiguration |
No server behaviour is attached to these today — no handler at all, or a handler that returns NotImplemented after the authorization check. Withholding them costs nothing and covers them in advance. |
Deliberately not withheld, and asserted by a test so that adding one is a deliberate act with a visible cost rather than an edit to a list:
PutBucketTagging,PutBucketEncryption,PutBucketNotification. These are bucket-level writes and an earlier draft did withhold them. None gives the caller access it does not already hold — the harm is to the owner’s posture, not to the access boundary — while a tenant handeds3:*onbucket/*and told the bucket is theirs may quite reasonably tag it, set default encryption, or wire up event notifications. Low security gain against a real compatibility cost is the wrong trade for a maintenance release.CreateBucket. It targets a bucket that does not exist yet, so there is nothing to mutate or destroy, and provisioning flows commonly create a tenant’s bucket with that tenant’s ownbucket/*credentials.- The read/list family (
ListBucket,GetBucketLocation, the configuration reads). Breaking these is what got upstream’s own attempt reverted. They wait for a migration-gated release.
Only Allow statements are affected. Deny statements keep the historical resource string, so no bucket lock is ever weakened, and NotResource exclusions keep their full reach.
Monotonicity, and the claim that was wrong twice
All of the above rests on one property: this change may remove permissions and must never add one. That property was asserted twice from reasoning rather than from tests, and was false both times. Recording how is more useful than recording only the final state.
The first attempt let the withheld slash reach the NotResource match as well — and NotResource is an exclusion. An Allow s3:* NotResource bucket/* statement historically did not apply to bucket-level requests on that bucket; matching the exclusion against the bare bucket name made it stop matching, so the Allow it qualified grew, for exactly the writes being protected.
The second attempt fixed that and shipped saying the result was provably monotone. An independent adversarial review of that release produced a counterexample. Withholding the slash does not merely remove a match — it changes which string patterns are matched against, and a pattern can match "mybucket" without ever having matched "mybucket/". A fixed-width wildcard is the clean case:
? matches exactly one character. Against the nine-character "mybucket/" it does not match, so this statement never authorized the bucket-level write. Against the new eight-character "mybucket" it does, so the hardening granted something the buggy matcher refused.
The fix is not another special case. On the protected path the matcher now requires both forms to match — the bare bucket name and the historical "bucket/". The result is an intersection with the historical decision, so it is monotone by construction: there is no pattern it can newly satisfy, and no argument left to get wrong. mybucket* still grants (it matched both all along), mybucket/* is still withheld, and mybucke? is refused exactly as it always was.
Two lessons are worth carrying forward. A correctness fix in an authorization path must never make anything newly allowed — and the only way to know is to test both directions, because the reasoning felt airtight in both cases where it wasn’t. And when a security property is load-bearing, build it out of an operation that cannot violate it rather than out of a case analysis believed to be complete.
Evidence
The property is verified rather than argued. A decision corpus of 27,000 authorization outcomes — 15 resource patterns × 3 buckets × 5 object names × 20 actions × 6 statement forms — was generated against both the pre-hardening baseline and this release and compared entry by entry:
| Transition | Count |
|---|---|
false → true (broadening) |
0 |
true → false (narrowing) |
144 |
| unchanged | 26,856 |
Every one of the 144 narrowed outcomes falls inside the design intent, with nothing outside it: exactly the twelve protected actions; only the three Allow statement forms, with zero transitions for Deny, NotResource-excluded or deny-NotResource forms; only four object-only resource patterns; and only bucket-level requests, with object-level requests entirely untouched. 12 × 4 × 3 = 144, fully accounted for.
Regression coverage exists at both layers. In this repository, twelve matcher tests pin each direction, including an invariant test that every protected action really is bucket-only — ResetBucketReplicationState, despite its name, is an object action and stays out. In the server, three end-to-end tests drive the real handlers at the client, inline-session-policy and S3-router levels; all three fail against the pre-fix build and pass against this one.
What to change
You are affected only if a stored policy grants one of the twelve actions — or s3:* — on a resource pattern containing /, with no bare bucket ARN for the same bucket. The fix is to add the bare ARN alongside the object pattern:
That pairing is the conventional form, is what upstream’s own tests use, and worked before this release as well. Built-in canned policies are unaffected — readwrite, readonly, writeonly and diagnostics all use Resource: "*".
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on, read once at startup, restores the historical matching in full — both the over-granting and the over-blocking. It is a single global switch; per-action scoping is deferred.
Policy Condition-Key Lookup Order
getValuesByKey() previously looked up a policy condition key by its canonical MIME spelling (http.CanonicalHeaderKey) before trying the original name. The map it reads mixes values calculated by the server for the current request (SourceIp, SecureTransport, CurrentTime, username and others, stored under condition-key spellings) with HTTP headers supplied by the request (stored under canonical MIME spellings).
Checking the canonical spelling first allowed a client header to override a value calculated by the server.
For a MinIO server this is a policy bypass. The simplest example is s3:prefix: a Prefix request header could satisfy a home-directory prefix condition while the real ?prefix= query parameter still listed the entire bucket. The same path reached aws:SourceIp, aws:SecureTransport, aws:CurrentTime, aws:EpochTime, aws:username, aws:userid, aws:principaltype, aws:UserAgent, aws:groups, ldap:username, ldap:groups, jwt:groups, s3:versionid, s3:signatureversion, s3:signatureAge, s3:authType and s3:LocationConstraint. Anonymous bucket policies were directly exposed. SigV4 did not prevent the attack because a client can add a header that is not listed in SignedHeaders.
There was a second consequence: when the server stored a value under one spelling and the policy key resolved another, the wrong entry won. s3:object-lock-mode could resolve to the caller’s X-Amz-Object-Lock-Mode header rather than the retention mode the server would actually apply.
The fix reverses the lookup order: match the condition key’s exact name first, then use the canonical spelling only as a fallback for condition keys that genuinely name request headers, such as the s3:x-amz-* family. This ports minio/pkg#226 and adds regression coverage the upstream change did not carry.
At the library’s raw-map layer, if a producer stores one logical field under both the exact condition name and its canonical MIME name, the exact name now wins. This is a library lookup rule, not an S3 wire-protocol rule that says query parameters take precedence. The Silo server first normalizes condition values by their real source. For storage class and upload tagging, where both Header and query forms remain compatible, Header presence wins, including an empty value; query is only the fallback.
LDAP Connection Path
Three defects in connect(). Two were introduced by this fork in b0c08a7 and shipped in v3.6.2 and v3.6.3. Users of either release should upgrade promptly.
StartTLS was skipped when ServerInsecure was enabled. Upstream called StartTLS in an outer block controlled only by ServerStartTLS, so enabling both options created a plaintext connection and then upgraded it. b0c08a7 moved the call into an else branch, making StartTLS unreachable whenever ServerInsecure was true. The connection stayed plaintext and the following bind sent credentials over it. MinIO exposes MINIO_IDENTITY_LDAP_SERVER_INSECURE and MINIO_IDENTITY_LDAP_SERVER_STARTTLS independently and Validate() rejects no combination, so this state was reachable.
This release restores the upstream semantics: the two switches are additive, not mutually exclusive. ServerInsecure disables implicit ldaps://; ServerStartTLS still performs the upgrade. The exposure window is limited to v3.6.2 and v3.6.3.
A Config without a TLS section could panic on the ldaps:// path. After l.TLS.Clone() moved outside the StartTLS branch, ordinary ldaps:// connections also called it. Clone() returns nil for a nil receiver, but the next line assigned ServerName. The MinIO server always supplies TLS settings, but this is a library and mc also consumes it. The code now falls back to an empty tls.Config, matching what DialURL would have built.
StartTLS had no deadline. go-ldap only starts its request timer when requestTimeout > 0, while StartTLS itself has no timeout. A server that completed TCP setup and then stopped responding to the extension request could hold the connect goroutine forever. The timer is now armed before StartTLS.
A failed StartTLS leaked the connection. Inherited from upstream. Dial failures do not return a connection, making StartTLS failure the only connect() path that could return both a connection and an error. Callers only took ownership when the error was nil, leaving a socket behind for every login attempt against a server with a broken upgrade. The failure path now closes the connection and returns nil.
Other Fixes
- certs: file watchers were never stopped.
Manager.AddCertificate()registered twonotify.Watch()calls and stopped neither: if the second failed, the first leaked, and both survived until process exit after the manager closed.Certificate.Watch()andwatchFile()had the same problem. All four paths now usewatchDirSafe(), which returns a stop function invoked on errors andctx.Done(). This ports thecerts/part of minio/pkg#228. On Windows the function replaces filesystem notification with polling rather than using polling only as a failure fallback, so certificate reload can lag by onesymlinkReloadInterval(10 seconds). This fork has no Windows CI; that platform was only cross-compiled. - rng: reader subkeys came from a zeroed local variable.
init()read 32 bytes of entropy intor.tmpbut derived four subkeys from a same-named zeroed local, collapsing four per-block streams into one.Reset()andResetSize()then replayed the previous stream byte for byte. MinIO creates a new reader for eachrandreader.New()call and never resets it, so the practical server impact is limited; warp exposed the defect. This ports minio/pkg#230. - xtime:
DurationimplementedUnmarshalJSONbut notMarshalJSON. Encoding produced an integer number of nanoseconds while decoding unconditionally stripped the first and last byte and expected a quoted string, so neither direction could round-trip. It now encodes usingtime.Duration’s string form. This ports minio/pkg#242.
Compatibility Impact
- Twelve bucket-level write actions are no longer authorized through an object-only resource pattern. See What to change. Object access,
ListBucket,CreateBucket, bucket tagging, default encryption and event notification are all unaffected, as areDenystatements andNotResourceexclusions. - The minimum Go version moves from
1.26.1down to1.25.0. A patch number in thegodirective is a hard minimum for every consumer, not a record of the toolchain used to build the module. The conventional split is a language version on thegoline and a development version on a separatetoolchainline.1.25.0is what the dependency graph actually requires and what upstream declares. CI builds the complete test suite with Go 1.25 underGOTOOLCHAIN=local, so the minimum is proven rather than aspirational. - The JSON wire format of
xtime.Durationchanges from a nanosecond integer to a duration string such as"2h"or"30m". Persisted numeric values can no longer be read back. No such use was found in MinIO ormc: batch job definitions persist as YAML and the msgp path remains int64. - Deployments with both
ServerInsecureandServerStartTLSenabled whose LDAP server does not support StartTLS connected successfully in plaintext on v3.6.2/v3.6.3 and now fail to connect. That is the correct result, but it surfaces during connection rather than configuration validation. DisableServerStartTLSfor such a server. Policy.IsAllowedActionscan disagree with a direct decision for the twelve protected actions. It enumeratesSupportedActions, which includes thes3:*pattern itself, so the returned set can contains3:*— and therefore appear to permit a protected action — while the direct evaluation denies it. Nothing in the server calls it, and Console calls it with an empty bucket name, which never reaches the hardened branch. Recorded rather than changed, because altering a public API’s output in a maintenance release is the larger risk.
Divergence from Upstream v3.11.0
The version number follows upstream’s line and makes no claim of identical content. The measured delta, comparing action-string constants across policy/:
| Count | |
|---|---|
Upstream minio/pkg v3.11.0 |
291 |
silo-pkg v3.11.0 |
270 |
24 actions exist only upstream: six s3:*ObjectAnnotation* actions, five admin: actions (DistJobStatus, Get/SetBucketCompression, two TablesReplication*), and thirteen s3tables: actions covering function CRUD and tagging. These belong to the AIStor vocabulary this fork deliberately does not carry, because the community server does not implement them.
Three actions are named differently on each side. Upstream renamed and split these; this fork retains the earlier names:
silo-pkg v3.11.0 |
upstream minio/pkg v3.11.0 |
|---|---|
s3tables:TagResource |
s3tables:TagTable, s3tables:TagWarehouse |
s3tables:UntagResource |
s3tables:UntagTable, s3tables:UntagWarehouse |
s3tables:ListTagsForResource |
s3tables:ListTagsForTable, s3tables:ListTagsForWarehouse |
A policy naming any of these six action strings therefore validates on exactly one of the two. Nothing in the Silo server, mc or Console references them, so there is no impact inside this ecosystem — but a consumer swapping upstream v3.11.0 for this release should know the vocabulary is not interchangeable.
rng has no arm64 assembly. Upstream added rng/xor_arm64.{go,s} after this fork’s divergence point; this release falls back to the pure-Go xor_noasm.go path on arm64. The result is correct and cross-compiles cleanly, but slower than upstream on that architecture. It is a clean candidate for a future sync, being a pure performance change with no vocabulary entanglement.
Companion Server Behavior
- The condition-key change in this release must be paired with the server changes that reserve internal condition-key names and populate values by semantic source, as noted at the top.
s3:signatureAgeis exposed only after the SigV4 presigned-request verifier calculates it. A client-suppliedx-amz-signature-ageHeader is ignored on every other request type.s3:prefix,s3:delimiterands3:max-keyscome only from query parameters. Content hash, copy source, metadata directive, SSE and object-lock conditions come only from the corresponding headers. TheX-Amz-Content-Sha256query value consumed while verifying a presigned request does not become a policy condition.s3:x-amz-storage-classretains its compatible query form, as do request tags onPutObjectandCreateMultipartUpload. For both fields, Header presence wins and query is used only when the Header is absent.s3:ExistingObjectTag/*comes only from tags loaded from the stored object, so a request’s ownX-Amz-Taggingcan no longer impersonate existing object state.PutObject,CreateMultipartUploadandPutObjectTaggingbinds3:RequestObjectTag/*to the tag input those handlers consume. Other action paths retain the historicalX-Amz-TaggingHeader fallback for compatibility, so treat request-tag conditions as constraints only where the API actually consumes tags.aws:SourceIpis calculated from forwarding headers. Whether it is enforceable depends on the server’s trusted-proxy configuration; see the server’s own release notes forMINIO_API_TRUSTED_PROXIES.
Verification
Everything below was run against the tagged commit, with the working tree clean and the tag pointing at HEAD:
make test— golangci-lint plusgo test -race -tags kqueue ./..., all packages passing.go mod tidy -diffclean;gofmt -lempty;go vet ./...clean.- Cross-compilation for
linux/amd64,linux/arm64,darwin/arm64andwindows/amd64. govulncheck ./...— zero reachable vulnerabilities. One module-level notice remains, GO-2026-5932 inx/crypto/openpgp; that package is unmaintained, has no fixed version, and this repository does not import it.- Resolution from an empty module cache through the public proxy, confirming the release is fetchable as published.
- The 27,000-outcome authorization corpus described above.
Dependencies and Tooling
Dependency updates clear nine reachable findings previously reported by govulncheck: seven x/crypto/ssh issues reached through sftp, GO-2026-6061 in gRPC reached through etcd, and GO-2026-4945 in go-jose reached through oidc.
Five dependencies — minio-go, minio/mux, etcd client/v3, go-oidc and lestrrat-go/jwx — were deliberately not upgraded. MinIO consumes this module through replace, and Minimal Version Selection chooses the highest version in the entire graph, so upgrading them here would also pull the server forward. None has a reported vulnerability requiring that change.
All three workflows previously asked setup-go for a Go version lower than go.mod required and failed on the first Go command; they are now aligned. The linter also fetched an installer from the master branch and reinstalled it on every run. The URL and version are now pinned to v2.11.3, and a matching installed version skips the download.
Changes Deliberately Not Taken from Upstream
- AIStor policy vocabulary (Memory/cortex, Tables/Iceberg, KMS, compression and annotations) and the typed action-constant refactor, none of which the community server implements. This is the source of the action-vocabulary delta.
securityAuditAdmin, which grantsadmin:ExportIAMand therefore exposes every secret key despite what the name suggests.- rng AVX2/NEON assembly. Revisiting the arm64 half is noted above as a future sync candidate.
net.BandwidthBytesPerSec(declared but never read upstream),replicationAdminandDistJobStatusAction.- Two changes initially taken and removed after review: the
consolereadonlybuilt-in policy andGetAllGlobalCertificates. Neither has a consumer. Once operators bind a built-in policy name to users, withdrawing it is particularly unsafe: policy mappings persist by name, and an unresolved name merges into an empty policy that denies everything. Its inheritedadmin:CreateUserDeny also cannot be combined withiamAdmin. The certificate helper inventoried a cache the community server never populates. - Upstream’s golangci-lint
tooldirective, which would add roughly 200 linter dependencies to every downstream consumer’s module graph.
Deliberately Deferred
The general problem in minio/minio#20449 — that bucket/* still reaches ListBucket, GetBucketLocation, the configuration reads, CreateBucket and the three tenant-plausible writes — is not closed here. Closing it means revoking grants real deployments depend on, so it belongs to a release that carries a migration path.
What that release owes operators is more than a longer action list, because no one can enumerate every deployment’s stored policies — which puts a hard ceiling on any approach that picks the protected set by guessing. Three things raise it:
- A startup policy audit that walks stored policies and names each one whose meaning changes, in both the grant and the deny direction. It is read-only and can ship before the enforcement change rather than with it, turning an upgrade surprise into a pre-upgrade checklist.
- A denial that explains itself. When a request is refused because only an object-scoped grant matched, say so and name the compatibility switch. A break an operator can diagnose in thirty seconds costs an order of magnitude less than a silent one.
- A switch with a scope.
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCHis all-or-nothing today, so an operator who needs one action back must reopen the self-escalation path along with it.
Related Commits
- d8b1fa7: fix(policy): settle the bucket-write hardening’s scope and monotonicity
- 1f97549: fix(policy): extend the bucket-write hardening to every bucket-only write
- 3c24ad1: fix(policy): withhold object-only grants from sensitive bucket writes
- da6a22a: docs: say what this fork is and how to depend on it
- 4055b2f: fix(xtime): marshal Duration as a duration string
- 13c26cd: fix(rng): initialize the reader subkeys from the seeded entropy
- 88b37ac: fix(certs): stop file watchers on every exit path
- 74dd36e: fix(ldap): keep StartTLS when ServerInsecure is also set
- 424c3d0: fix(ldap): close the connection when StartTLS fails
- 045d10f: fix(ldap): guard a nil TLS config and arm the StartTLS deadline
- 5c4bf50: fix(policy): prefer the exact key name over the canonical header form
- 802539f: chore(deps): refresh the dependency set and declare the real minimum Go
- e4ec64a: ci: build on the Go version go.mod requires, and prove the declared minimum
- 747d8b8: build: pin the golangci-lint installer and skip a matching install
2.13 - Silo 20260903 Released
Status on 2026-09-13: this remains the latest published Server; its dependencies have not changed. Later security, storage and dependency fixes are on main, including the unreleased SN-2026-011 fix. New standalone mcli/pkg releases do not replace this artifact’s contents; see the component matrix.
Version date: 2026-09-03 · Published: 2026-09-04 · Version: RELEASE.2026-09-03T13-18-01Z · Previous release: RELEASE.2026-08-06T00-00-00Z
SILO 20260903 is the security and correctness release following the first fully rebranded Silo release. Its main theme is not a new delivery surface but a tighter set of server invariants: client headers do not create replication authority, browser origins cannot trigger bucket-metadata I/O before authentication, concurrent bucket-configuration writers cannot silently overwrite one another, and authorization and checksum behavior more closely match AWS S3.
This note deliberately uses 20260806 as its only release baseline. Intermediate component builds remain identifiable in their source repositories, but the public documentation below consolidates the complete server, Console, shared-package, and client result delivered on 2026-09-03.
Highlights
- Replication trust is authorization-derived.
X-Minio-Source-Replication-Requestand related internal fields gain replication semantics only after signature verification, an exact marker check, ands3:ReplicateObjectors3:ReplicateDeleteauthorization. Untrusted fields are stripped after authentication so SigV4 remains valid. - Per-bucket CORS is complete.
PUT,GET, andDELETE ?corspersist a real bucket policy, override the server-wide fallback, and converge through site replication. The pre-authentication lookup is resident-only and remains fail-closed during startup and after a known metadata-load failure. - Bucket metadata updates are serialized. A shared
metadata.lockcovers all whole-record configuration writers, migrations, imports, site adoption, and healing.ForceCreateno longer erases existing configuration, and an Object Lock document always implies plainEnabledversioning. - Authorization matches the requested operation. Explicit version deletion requires
s3:DeleteObjectVersion; user and group enable/disable calls require the action matching the requested status; policy writes reject empty ARN namespaces. - Checksum behavior moves toward AWS S3. Upload parts can receive server-computed checksums, federated
UploadPartCopyreturns its checksum, multipart completion reportsChecksumType, invalid assertions use AWS-compatible failures, andCRC64NVMEwithCOMPOSITEis rejected. - SSE-C reads and copies authenticate the key consistently. Zero-byte objects and
GetObjectAttributesno longer bypass key validation; null-version rewrites and in-place key rotation no longer leave unreadable ciphertext or inconsistent checksum metadata. - The release was simplified before publication. Dead code and obsolete lint exemptions were removed, compatibility snapshots now track served surfaces instead of source-level noise, dependencies were pinned to reviewed revisions, and flaky timeout tests received a private random source.
- The client release is part of the same baseline. mcli 20260903 adds read-only checksum verification, fail-closed credential redaction, repaired JSON metrics and S3 Select behavior, strict policy writes, signed packages, provenance, and verified amd64/arm64 images.
Security fixes
The repository security ledger assigns stable local identifiers to five findings without CVEs. An SN- identifier is not a CVE. The complete threat models, compatibility effects, and operator actions are collected in SILO 20260903 Security Notes.
| ID | Affected surface | Fixed invariant | Operator impact |
|---|---|---|---|
SN-2026-006 |
SSE-C reads and copies of zero-byte objects | The supplied customer key is authenticated even when there is no payload block to decrypt | Wrong keys now return 403 AccessDenied; no change for correct keys |
SN-2026-007 |
GetObjectAttributes on SSE-C objects |
Object attributes require the customer key, except for an actually authorized replication peer | A bare replication marker cannot skip key authentication |
SN-2026-008 |
Internal replication fields across reads, writes, multipart, deletes, Snowball, and events | Replication semantics require authentication plus the matching replication permission | Ordinary clients carrying internal-looking headers keep ordinary S3 semantics |
SN-2026-009 |
Admin user and group status changes | Enable and disable are authorized against different actions | Split custom admin policies that previously granted only one action but relied on both |
SN-2026-010 |
DeleteObject / DeleteObjects with versionId |
Explicit version deletion requires s3:DeleteObjectVersion |
Review grants and add an explicit Deny where Deny s3:DeleteObject was intended to block permanent deletion |
All five defects were inherited from the archived upstream server lineage and affect every earlier SILO release. SN-2026-008 completes the receiver-wide repair begun by CVE-2026-34204.
The embedded Console v2.3.0 line also contains a trusted-proxy boundary, TLS verification repair, response and log redaction, and bounded WebSocket forwarding. The standalone Console release and the server embedding remain separately verifiable artifacts.
S3 correctness and interoperability
Multipart checksums
This release closes the multipart interoperability group reported in #46, #47, #48, and #50:
UploadPartcomputes the selected checksum when the client selects an algorithm at upload creation but omits a per-part checksum header.UploadPartCopyreturns the remote checksum through a legacy federated backend instead of dropping it.CompleteMultipartUploadincludesChecksumTypeand uses AWS-compatible error codes when a checksum is absent, malformed, contradictory, or unsupported.- Unknown checksum algorithms and the invalid
CRC64NVME+COMPOSITEcombination are rejected rather than silently canonicalized.
The fixes are deliberately layered. The wire parser rejects invalid declarations, the multipart state records the selected algorithm and type, and completion validates the final claim against that state. Ordinary clients that do not opt into additional checksums are unchanged.
The individual evidence records are server-computed UploadPart checksums, multipart completion error semantics, and ChecksumType propagation.
CopyObject and SSE-C
CopyObject now:
- calculates checksums over the logical object before optional compression;
- returns the checksum fields in its response;
- preserves transform state on metadata-only copies;
- decrypts source checksum metadata with the source key and rewrites it for the destination key;
- handles null-version copies in either pool direction without losing current-version metadata;
- re-encrypts an in-place SSE-C key rotation when the object layer must rewrite the data.
The zero-byte and attributes fixes remove two independent places where “no data was decrypted” had accidentally been treated as “the key was authenticated.” The new rule is explicit: access to SSE-C metadata or ciphertext requires a successfully unsealed object key, unless the request has already crossed the authorized replica boundary.
The CopyObject state matrix and live interoperability evidence are preserved in CopyObject SSE-C, checksum, and response correctness.
Listing and federation
- ListObjects shortcuts now return
NoSuchBucketfor a missing bucket even when a prefix selects an early-return path (#32, PR #37). - Federated
UploadPartCopypreserves the backend checksum in the public response (#72). - Site-replication status counts metadata operations per site instead of multiplying a group-wide result, and validates CORS as an independent configuration field.
The missing-bucket shortcut is documented in ListObjects must prove the bucket before it optimizes.
Per-bucket CORS and request trust
Bucket CORS was implemented end to end rather than as a handler-only shim:
- the S3 XML type and rule matcher enforce the protocol grammar and multi-rule preflight behavior;
- bucket metadata stores the raw configuration and returns it byte-for-byte on
GET ?cors; - the outer middleware uses the bucket policy when present and the global origin setting only as fallback;
- site replication carries CORS in its own tombstone-aware last-writer-wins register;
- pre-authentication lookup never loads metadata or creates cache entries;
- startup, a known bucket-metadata load failure, deletion, refresh, and on-demand reload all update the fail-closed state explicitly.
The load-failure state is intentionally retained. A pre-signed URL authenticates itself without consulting bucket policy; if the server forgets that a real bucket’s CORS document failed to load, falling back to a permissive global policy would discard the only browser-origin boundary around that otherwise authorized request.
Replication-request hardening shares the same principle. The server authenticates the original request first because internal headers may be covered by SigV4, derives one private trust decision from the authenticated identity and permission, and only then sanitizes an untrusted request clone. The clone shares the original trailer map so streaming checksums remain visible. Snowball workers derive trust per entry rather than inheriting a request-wide privilege bit.
The full design and rejected alternatives are preserved in No I/O Before Auth, No Privilege From Headers.
Bucket metadata and Object Lock
Bucket configuration is stored as one .metadata.bin record but was previously updated through independent locks for policy, lifecycle, encryption, tags, quota, replication, Object Lock, and CORS. Two valid read-modify-write operations could therefore each succeed while the later save silently discarded the earlier field.
This release introduces one bounded metadata.lock transaction around every whole-record mutation. The same lock covers:
- normal configuration saves and deletes;
- legacy metadata migration and import;
- bucket creation,
ForceCreate, and site adoption; - healing and inconsistent-version repair;
- replication receive paths that merge changed fields.
The implementation does not turn the metadata system into a generic transaction framework. It keeps the existing record and parsers, centralizes only the serialization boundary, and narrows replicated updates to fields that actually changed.
Object Lock receives a second invariant. Once a valid lock configuration parses as enabled, versioning is normalized to a plain Enabled document; prefix exclusion and suspended states cannot survive. The decision is based on the parsed configuration, not byte equality with the smallest possible XML document, so a valid Default Retention rule cannot bypass it. Update, read-back, and disk-reload tests cover both malformed prior versioning shapes.
IAM, policy, and configuration behavior
- Explicit version deletion follows AWS
s3:DeleteObjectVersionauthorization while replication deletes retains3:ReplicateDelete. SetUserStatusandSetGroupStatusauthorize the requested target state instead of always checking the enable action.- New and updated named policies and service-account policies reject bare S3, S3 Tables, and KMS ARN namespace prefixes, their historical serialized forms, and statements that combine
ResourcewithNotResource. Stored policies continue to load. - Enabled legacy PostgreSQL or MySQL notification targets must supply a connection string. Startup now fails with a credential-free diagnostic instead of silently discarding all targets.
MINIO_CONFIG_ENV_FILEparsing preserves named targets, quoted values, and supported shell-like assignments without evaluating the file as shell code.
These changes are compatibility tightenings, not format migrations. Review the upgrade checklist before deploying.
Detailed records cover target-state user/group authorization, safe configuration environment-file parsing, and fail-fast legacy database notification migration.
Components
The 20260903 dependency and companion-release line is pinned as follows:
- Go 1.27.1
minio-go/v7: upstream-compatible pseudo-version ending in0e78d3f18efe; the temporarysilo-gofork is retired from the server graphmadmin-go/v3:v3.0.110silo-pkg/v3:v3.13.2, under its owngithub.com/pgsty/silo-pkg/v3module path- embedded Console:
v2.3.0, including the trusted-proxy, TLS, redaction, and WebSocket security work - bundled mcli:
RELEASE.2026-09-03T07-13-05Z, with themccompatibility alias preserved - Helm chart:
7.0.2; use the published server imagepgsty/silo:RELEASE.2026-09-03T13-18-01Zand the mcli imagepgsty/mc:RELEASE.2026-09-03T07-13-05Z
Tagged-source Helm note: the signed Server tag retains the provisional
RELEASE.2026-09-03T00-00-00Zvalue inhelm/silo. Post-release commit4c164907fcorrects the currentmainchart to the published tag. If you render Chart 7.0.2 directly from the tagged source archive, setimage.tag=RELEASE.2026-09-03T13-18-01Zexplicitly; binary and package artifacts are unaffected.
The server deliberately preserves the MinIO-compatible Go module and wire identifiers. Replacing an import target does not rename the public S3/Admin API, MINIO_* configuration, x-minio-* headers, /minio/* routes, or on-disk metadata.
Engineering cleanup
The final release pass removed complexity where it did not protect a compatibility or correctness boundary:
- removed dead encryption helpers, obsolete handler paths, and unused event-target functions;
- replaced a source-level exported-symbol inventory with the smaller served-route compatibility baseline;
- removed the
wait_pipelint exclusion and migrated togomodguard_v2; - consolidated CORS load-failure lifecycle changes behind two helpers instead of open-coding set operations in six paths;
- narrowed site-replication imports to changed metadata fields;
- gave dynamic-timeout tests their own random source so parallel tests cannot mutate a package-global seed;
- retained the shared metadata lock, CORS tombstones, two-level replication trust, and adversarial tests because each protects a reproduced failure rather than a hypothetical abstraction.
The complete adversarial review, including the fixes found after the first “ready” claim, is recorded in SILO Server 20260903 Pre-release Review.
Upgrade checklist
Read Upgrading from RELEASE.2026-08-06 before changing production. At minimum:
- grant
s3:DeleteObjectVersionto identities that intentionally delete explicit versions; add its Deny next to anyDeny s3:DeleteObjectintended to prevent permanent deletion; - split custom admin grants for enable and disable operations;
- correct bare ARN prefixes before re-submitting policies;
- add connection strings to enabled legacy PostgreSQL/MySQL notification targets;
- verify applications that explicitly choose checksum algorithms or types;
- upgrade every member of a site-replication group before creating or changing per-bucket CORS;
- snapshot CORS configuration before rollback, because 20260806 does not understand it;
- switch every node of a distributed cluster to the same binary in one maintenance operation; mixed Silo binaries do not form a cluster.
The on-disk object and erasure formats are unchanged. This does not make a mixed-version cluster supported.
Known issues and explicit deferrals
These items are not fixed by this release:
- Conditional delete (#10, PR #12).
DeleteObjectignores HTTPIf-MatchandDeleteObjectsignores each<Object><ETag>. Both delete unconditionally. A reviewed single-object repair exists outside this release, but batch semantics and the original PR are incomplete; shipping a partial condition contract would be riskier than the documented deferral. - Non-CORS multi-site configuration deletion (#77). A peer can restore a policy, SSE, tag, or quota configuration deleted on another site. Single-site deployments are unaffected. CORS is not affected because it uses a tombstone-aware register. Deployments relying on these replicated deletions must verify every site.
ListMultipartUploadsfilters (#79).prefixbehaves like an exact key match, whilemax-uploads,key-marker, anddelimiterare not fully honored; see the conformance analysis.- Legacy federation
CopyObject(#99, #100). The legacy federated backend can ignore a requested checksum algorithm and rejects inline source objects. Deployments not using that backend are unaffected. - Mixed-version site replication. A 20260806 peer accepts but ignores bucket CORS and continues reporting a mismatch. Do not configure CORS until all sites are upgraded.
- Rollback of bucket CORS. 20260806 drops the unknown CORS field if it rewrites the bucket record. Export the configuration first and recreate it after upgrading again.
Open enhancement work such as ILM relocation (PR #60), broader SSE support (#61), operator discovery (#30), NATS target hot reload (#40), and Renovate (#20) is outside this release’s production-safety boundary.
Verification and evidence boundary
The complete local acceptance was run on ebac0ca73bbf251b070bb6df4d8005015841f901:
- full
cmdandinternaltest suites; go test -race ./cmd, passing in 365.448 seconds;- lint with 0 issues, the rebrand/compatibility guard, generated-file checks, and
govulncheck; make verifyacross FS, erasure, distributed erasure, erasure sets, multi-pool, and IPv6 multi-pool: 174 PASS / 0 FAIL.
One subsequent one-line fix, 84e1580a4, clears the CORS load-failure bit after a successful on-demand GetConfig reload. Later dependency work moves the toolchain to Go 1.27.1 and x/crypto 0.56.0. On exact release source 9b11dc9469e650815b775cb47b039610644f5da4, git diff --check, full cmd and internal tests, the complete cmd race suite, rebrand and generated-file guards, lint, govulncheck, and remote Go CI, VulnCheck, Test Release, and Release workflows pass.
The final functional acceptance used four Ubuntu 24.04 arm64 VMs with four XFS data disks each. It covered a coordinated 20260806 → 20260903 upgrade, rollback, and second upgrade; the complete mcli functional script over TLS; 1,004-object upload/download checksum comparison; one-node and quorum-loss behavior; single-drive reconstruction; and two-site replication of objects, versions, CORS, tags, quota, ILM, Object Lock, IAM, delete markers, and offline catch-up. The cluster finished with all 16 drives online and no unrecovered data mismatch.
Container publication exposed one delivery-only pin error after the binary/package Release was public: the first Docker workflow stopped before pushing any image because the tagged Dockerfile selected mcli 20260903 but retained the two 20260901 archive digests. Commits ce2326c94 and 9936a69d8 corrected those pins and added a narrow recovery mode that resolves them from the immutable, attested mcli checksum manifest. Recovery run 33847571968 then built, ran, SBOM-attested, and published both architectures from the original Server release archives. The classic release/latest manifest is sha256:b616a0cf8cb281e7e6bb3c9b1fb53875b4016a2878223925541c18f82d6c5ca3; distroless is sha256:2711cdc5bce0ada353a4871dcda60f48f46414fdd69a45c36c71f6b7ebe1aead.
Delivery remains independently checkable rather than inferred from those source tests. The published server GitHub release identifies the exact tagged source and verified archives, packages, checksums, SBOMs, and provenance; its classic and distroless images publish matching amd64/arm64 manifests. The separately published mcli release is immutable, contains 19 verified assets, and publishes matching amd64/arm64 release and latest images. Shared package v3.13.2 and Console v2.3.0 retain their own release evidence.
Resolved issue ledger
The tracker was reconciled during release preparation. Core issues are closed; deliberately excluded work remains in its own open issue:
| Tracker | Resolution |
|---|---|
| #102 | Closed by merged PR #103: shared metadata.lock and complete writer/migration/adoption coverage; reproduction-first residual audit split to #105 |
| #58 | Closed by merged PR #104: explicit-version authorization with multi-delete context preservation and least-privilege replication tests; PR #59 closed as superseded |
| #46, #47, #48, #50 | Multipart checksum selection, computation, response, validation, and errors |
| #32, PR #37 | Missing-bucket listing shortcut returns NoSuchBucket |
| PR #57 | CompleteMultipartUploadResult.ChecksumType is present in the selected client/server stack |
Selected changes
1c9a2431fthrough13e6458d9: implement per-bucket CORS and site replication938603458through04b097fd9, plus Snowball follow-ups: authenticate replication semantics and remove pre-authentication metadata I/Of9f9fa6c9through32a1b81e4: reproduce and serialize cross-type bucket metadata updates3b5de82f5,21646eebd: enforce Object Lock’s versioning invariant, including retention-rule documentsf8b598f1dthroughd2d47a41f: align explicit-version delete authorizationb73581b05,474cd5801,74c97d005: authenticate SSE-C zero-byte and attributes reads7fea6d5a5,5d152416d,7e079ff05,d28885d0e: align multipart checksum behaviorc0e715977,e73436c99,ffb70eb37: repair CopyObject transforms, checksums, and SSE-C rotation84e1580a4: clear the load-failure guard after an on-demand metadata reload succeeds
Acknowledgments
This release incorporates reports and proposed fixes from community contributors as well as the fork’s security and compatibility audit. In particular, the checksum, missing-bucket listing, and explicit-version authorization work started from public issues and pull requests. The release’s final shape also reflects adversarial review: findings were reproduced before repair, inherited limitations were separated from release regressions, and validation claims were rerun against the tree they describe.
The complete authorship record remains in CONTRIBUTORS.md.
2.14 - Explicit Version Deletes Now Require DeleteObjectVersion
Release status: this change is implemented in pgsty/silo#104, tracking issue #58. Publishing this note does not by itself mean that a server release, package, image, or deployment contains the change.
SILO now maps object-delete authorization to the operation the request will actually perform:
| Request | Required action |
|---|---|
No versionId |
s3:DeleteObject |
| Version UUID | s3:DeleteObjectVersion |
Explicit versionId=null |
s3:DeleteObjectVersion |
DeleteObjects |
The mapping is applied independently to every XML entry |
Previously, SILO required s3:DeleteObject for every case and used
s3:DeleteObjectVersion only as an explicit-deny check. A principal holding
only DeleteObject could therefore permanently remove a named historical
version. Conversely, a least-privilege purge principal holding only
DeleteObjectVersion could not perform the operation it was intended for.
After this change, a DeleteObject-only principal can still perform an
unversioned delete or create a delete marker, but receives AccessDenied for a
named UUID or null version. A DeleteObjectVersion-only principal can remove
the named version but cannot create a delete marker. Explicit denies and
s3:versionid conditions retain normal policy precedence. Multi-delete uses
each entry’s VersionId; a query-level decoy cannot change another entry’s
condition value.
Replication compatibility
Bucket and site replication target policies do not need to add
s3:DeleteObjectVersion. An authenticated request earns replication delete
trust through an exact internal marker and s3:ReplicateDelete; the receiver
then preserves the deployed s3:DeleteObject + s3:ReplicateDelete contract.
An explicit deny on s3:DeleteObjectVersion continues to block a replicated
version purge.
This distinction prevents an upgraded target from silently rejecting permanent
delete replication and also avoids granting new delete capability to a
ReplicateDelete-only credential. A real two-site regression used a target
user with the documented minimal policy and no DeleteObjectVersion; both a
permanent version delete and a delete marker converged successfully.
Upgrade impact
- Review user, service-account, OPA, and external authorization policies that
currently grant only
s3:DeleteObjectbut performmc rm --version-id,mc rm --versions, Console delete-all-versions, or SDK deletes withversionId. - External authorization plugins now see one
s3:DeleteObjectVersiondecision for an ordinary named-version delete instead of the former deny-only check followed bys3:DeleteObject. X-Minio-Force-Deleteprefix cleanup remains gated bys3:DeleteObject; it is not an explicit-version S3 request.- There is no wire or storage-format migration. Rolling back restores the old authorization mapping but does not alter stored objects or metadata.
See Object Deletion for the operator-facing permission matrix.
2.15 - mcli 20260903 Released
Published: 2026-09-03 · Version: RELEASE.2026-09-03T07-13-05Z · Source: a2ef95c0
Twenty-eight days after mcli 20260806, this release carries everything the client accumulated since. The toolchain moves to Go 1.27.1 and the shared dependency stack is aligned with the SILO family, with silo-pkg imported under its own module path so go.mod says what the binary is built from. The first genuinely new command since the fork arrives — mcli checksum verify, a read-only audit of stored S3 additional checksums against the bytes an endpoint actually returns. The panic that made --json admin prometheus metrics unusable is fixed, as is the response double-close that could panic after mcli sql had printed valid rows. And every path that can print a credential — --debug traces, admin trace, error messages, JSON error documents, checksum reports — was rewritten fail-closed after adversarial review. The release process changed as much as the code: artifacts come from a GPG-signed tag at the exact tip of main with green CI for that commit, ship LICENSE, NOTICE, and a generated CREDITS in every format, carry Sigstore build provenance, and land in an immutable GitHub release.
The release was built from a2ef95c0, the merge of pgsty/mc#27: 58 reachable commits, 50 of them non-merge commits, across 262 files (+12,742 / −6,566 lines) since RELEASE.2026-08-06T00-00-00Z. The intermediate 20260901 build remains available on GitHub, but this page treats 20260806 as the public documentation baseline and records the complete 20260903 result.
Behavior changes
- macOS 13 is the new floor. Go 1.27 links darwin binaries with a minimum OS version of 13.0; the darwin archives no longer run on macOS 12 and earlier.
mcli --json admin prometheus metricsworks again and returns metric families in deterministic name order. Plain-text metrics output is unchanged.pipewith empty standard input now performs a regular zero-byte PUT instead of a one-part multipart upload. The resulting ETag isd41d8cd98f00b204e9800998ecf8427erather than the<md5>-1form. Non-empty input is unchanged.- Policy writes are validated strictly on the client —
admin policy create,admin user svcacct add/set,idp ldap accesskey create/edit. A bare ARN such asarn:aws:s3:::is rejected (writearn:aws:s3:::*to mean every resource under that prefix); a named policy withoutVersionis rejected; an empty policy is rejected; mixing S3 and admin actions in one statement is rejected. The last three were already refused by the server, so only bare ARNs change what can be stored. Reads of existing policies remain permissive. Session policies for service accounts may still omitVersion, andadmin user svcacct set --policystill accepts an empty document — that is the way to clear an inline policy and return the account to its inherited one. - A malformed global flag before the command name is now an error.
mcli -H "bad" ls,--resolveand--limit-*values that fail to parse at the app level used to be silently ignored and the command ran without them. They now fail with the same message the command-level form produces, reported once on stderr — or as a JSON error document under--json— where the CLI library used to echo the text to stdout first. - Error messages no longer echo secrets.
alias setreports only the length requirement for a bad secret key;MC_CONFIG_ENV_FILEparse errors name a line number instead of repeating the line; SSE-C key errors withhold the key;admin config setmasks secret-bearingkey=valuepairs; remote replication targets are printed without credentials; a malformed--custom-headerentry is reported by position, never by value. - Support artifacts are private. Files written by
support diag,support profile,support perf,support inspect,admin cluster bucket export, andadmin cluster iam exportare created with mode0600, including the rotated backup of a previous file. They were0644or0666before. --debugshows less. AnAuthorizationheader keeps only its scheme (AWS4-HMAC-SHA256 **REDACTED**); tokens, cookies, API keys, SSE-C keys and every--custom-headervalue are redacted; 2xx response bodies are no longer dumped.admin traceoutput is redacted the same way in both its default and its verbose rendering: header values by name, query strings, bodies, error text and annotations.- The User-Agent reports
minio-go/v7.3.0(the SDK’s version constant) instead ofminio-go/v7.0.98.
Major Changes
- Go 1.27.1 across the module, container build stage, Linux/macOS/Windows CI, cross-builds, tidy checks, and vulnerability analysis. The Go-maintained module family is current:
x/net0.58.0,x/sys0.47.0,x/term0.45.0,x/text0.41.0,x/crypto0.56.0,x/mod0.40.0,x/sync0.22.0, andx/tools0.49.0.golangci-lintstays outside the product module graph and is installed independently at the pinned v2.13.1. - The SILO package under its own module path.
go.modnow requiresgithub.com/pgsty/silo-pkg/v3v3.13.2 directly. The client’s policy validation depends on helpers that exist only there —policy.ParseConfigStrictandpolicy.Resource.IsBareARN— and areplaceof upstreamminio/pkgwould not be inherited by projects consuming this module (pgsty/mc#12), so the requirement is declared honestly. Upstreamminio/pkgsurvives only as an indirect dependency ofcolorjson, held at v3.6.1 by the dependency-floor check, andgomodguardblocks the upstream import path from returning.minio-gomoves from v7.0.99 to the latest upstreamminio/minio-gomaster commit, represented byv7.3.1-0.20260828014306-0e78d3f18efe; the interimpgsty/silo-gofork is retired. The final refresh includes etcd 3.7.1,go-pretty6.8.3,klauspost/compress1.20.0, Prometheus client/common/procfs 1.24.1/0.71.0/0.22.0,grpc-gateway2.30.0, gRPC 1.83.2, Protobuf 1.36.12, and JWX 3.2.0.go-systemddeliberately stays at 22.6.0 through SILO’s portability pin — 22.7.0 does not compile on NetBSD. - Runtime version floors are guarded. A first dependency pass let the removal of development tools silently lower a dozen runtime selections — among them
shoenig/go-m1cpu, whose older releases crash during initialization on newer Apple Silicon when CGO is enabled — and was rejected.buildscripts/check-dependency-floorsnow compares every requirement ingo.modagainst the previousRELEASE.*tag and fails the build on a regression; the only permitted exception is thego-systemdpin. mcli checksum verify(design). Verifies stored S3 additional checksums — CRC32, CRC32C, CRC64NVME, SHA1, SHA256, of typeFULL_OBJECT— against the logical object bytes returned by the object API, for one object, a prefix with--recursive, all versions with--versionsor an exact--version-id, or a JSON Lines candidate manifest. It reads withIf-Matchon unversioned objects and re-stats afterwards, so a write that races the verification is reported asUNKNOWN_OBJECT_CHANGEDrather than as a mismatch. Each object yields one ofMATCH,MISMATCH,NO_CHECKSUM,WOULD_VERIFY(dry run), one of tenUNKNOWN_*classifications, or one of threeSKIPPED_*results; the summary carriesobjects, averifiedcount (MATCH+MISMATCH, the only results that actually recomputed a checksum), per-status counts andincomplete.--fail-onacceptsmismatch,unknown,no-checksum(fails when any object lacks a checksum or when nothing was verified at all),any(default: mismatches, unknowns, and objects skipped by--max-size), ornone.--reportwrites the same records to a new0600JSON Lines file. Results go to stdout whether or not stdout is a terminal, and with--jsonthey are compact JSON Lines outside a terminal — the non-TTY silence reported as pgsty/mc#5 is fixed. Dry runs,--max-workers,--max-size, time filters, and SSE-C through--enc-care supported. The command is verified against a real SILO server in CI.- Credential redaction, fail-closed. Three rounds of adversarial review by Codex showed that the original redaction — and its first two rewrites — recognized the shapes it expected and let everything else through: access keys containing a slash, an
Authorizationvalue whose first field is empty so the second one hides, non-hex signatures, secrets tucked into the credential scope orSignedHeaders,Proxy-Authorization, cookies,X-Api-Key, response trailers, a redirectLocationwith userinfo, a server that reflects request headers into an error body, a JSON-escaped secret inside--jsonerror output, a short token that appears twice. The final design has three rules and no exceptions. AnAuthorization-class value keeps only its scheme. A header whose name looks like a secret, or that the caller supplied through--custom-header, is redacted without inspecting its content. Text the client did not generate itself — response bodies, server error messages, trace notes — is scanned for credential shapes and then scrubbed of every secret literal the process has learned. That last layer is backed by a process-wide secret registry: alias secret keys and session tokens,MC_HOST_*values, SSE-C keys in every encoding, tier and service-principal secrets, GCS credential files, LDAP passwords and the STS credentials they return, SUBNET tokens,--secret-key/--account-key/--api-keyflags, and secret-bearingkey=valuearguments are registered as they are read, before validation and before any network call. Occurrences are collected as intervals and merged before replacement; JSON error documents are scrubbed on the decoded value tree, string leaves only, so keys and compact non-terminal formatting survive. Forty-odd adversarial cases, a black-box test that runs the real client against an endpoint that reflects everything it receives, and live captures against a SILO server pin the result. admin traceredaction. Server-supplied header maps for other clients’ requests take a path the S3 tracer never touched. Both the human-readable and the JSON rendering, in the default and the verbose form, are now produced from redacted copies — the event is shared between them and never mutated. Query strings (a presigned URL’s signature, a session token in the first parameter), bodies, error text and annotations are scrubbed with the event’s own header values, so a token another client sent cannot reach privileged trace output. Headers are redacted by name: a--custom-headeranother client sent under a name that does not look like a secret is shown as the server saw it.
Fixes
- JSON metrics no longer panic.
mcli --json admin prometheus metricsfailed for both the v2 and v3 metrics APIs withpanic: Invalid name validation scheme requested: unsetin the 20260806 client.madmin-gobuilt the zero value of the Prometheus text parser, whichprometheus/common0.66 and newer reject. The client now constructs the parser with UTF-8 validation, returns ordinary parse errors instead of panicking, and emits families in name order. pipewith empty input performs a regular PUT; the one-part multipart upload it used to issue for zero bytes was needless and produced a multipart-style ETag.- RPM payload validation aligned with the signing script, so an RPM whose contents do not match the release metadata is refused before it is signed.
sqlno longer panics after printing its rows. The SDK’s select reader closes the HTTP response itself on every terminal event, and the client closed it a second time; on a zstd-compressed response the two drains raced and about two in five queries against a SILO server ended inpanic: invalid memory addresswith exit status2after the rows had already been written (pgsty/mc#23). The client now cancels the request and drains the result pipe instead of closing the response again; 820 consecutive queries on a live server ran without a panic where the previous client failed 21 of 50.
Hardening
- Operator-produced files are no longer world-readable.
support diagwarned that its archive may contain sensitive environment information and then wrote it0666; the sharedmoveFilehelper relaxed every0600temporary file to0644on its way to the final name, and a0644left by an earlier client was carried into the rotated backup even when source permissions were preserved. Every support artifact is now created0600and chmodded before the first byte is written. Test_FullSuitefails on panic. The functional suite recovered from panics to run cleanup and then reported success; a panic anywhere in the suite, including in the binary build, was invisible to CI.- A leftover
play/alias in the new command’s help — the only file in the repository still advertising MinIO’s operated demo alias — is replaced, and the brand gate now rejects bare aliases in command examples, not only hostnames. - AGPL notices completed on six fork-authored files that carried only the first half of the license text.
Engineering and Delivery
- Third-party notices in every artifact.
CREDITSis generated bybuildscripts/gen-credits.shfrom the union of the runtime modules linked into all six GoReleaser targets and is compared in CI so a dependency change cannot leave it stale.LICENSE,NOTICE, andCREDITSship in the GoReleaser archives, the DEB/RPM/APK packages, and every container image under/licenses/. - CI runs on
main.GoandCrosscompileran only on pull requests, so the merge commit — the one that gets tagged — was never tested; both now run on every push tomain. Their concurrency group was keyed ongithub.head_ref, which is empty outside pull requests, so every manual run cancelled every other; it is now keyed on the pull request number with the ref as fallback.VulnCheckalso runs weekly, because a fork of an archived upstream can go weeks without a commit. The functional suite in CI now targets SILORELEASE.2026-08-06T00-00-00Z. - Release only from signed tags. The artifact workflow has no manual trigger; artifacts come from a pushed
RELEASE.*tag, and the event SHA, the commit the tag points to, and the checkout must all agree.verify-release-tag.shrequires an annotated tag with a valid OpenPGP signature from the expected key — fingerprint and key location come from repository variables rather than from the tree the tag points at, and the imported key must match the expected fingerprint.check-release-commit.shrequires the tagged commit to be the exact tip oforigin/mainand demands a successful push-to-mainrun of four workflows for that SHA, matched by workflow file path; pull request and manual runs are not evidence.check-release-state.sh(pgsty/mc#6) runs serially per tag, replaces exactly one existing Draft from scratch, refuses a published release, and re-checks for a single Draft before packages are uploaded. Every one of these decisions lives in a shell script with fixture coverage; the fakeghin the tests pins both the verdicts and the arguments passed, verbatim. - Provenance and immutability. Archives, checksums, DEB and APK packages carry Sigstore build provenance (
gh attestation verify <file> --repo pgsty/mc). RPMs are excluded on purpose:sign-release-rpms.shrewrites their bytes with a detached GPG signature from key9592A7BC7A682E7333376E09E7935D8DB9BD8B20while the release is still a Draft, and an attestation made earlier would name a digest no published RPM has. GitHub releases are immutable once published; fixing an asset means a new tag. Thepgsty/mccontainer image is built only from attested archives, aftergh attestation verifybinds each one torelease.yml@refs/tags/<tag>, the release commit digest, GitHub-hosted runners, its own filename and its SHA-256; the unpacked binary’s ELF machine type must match the archive’s architecture; each per-architecture image is pulled, run, and checked for the tag before the manifest is assembled, andlatestis written only from the two-architecture manifest. The UBI base image is pinned by digest and kept current by Dependabot. - Repository governance matches a single-maintainer project. Direct pushes to
mainare allowed; the stale pull-request ruleset and its exact-name status-check coupling were removed after this release. Release integrity does not depend on that branch rule: the artifact gate still requires successful push-to-mainGo, Crosscompile, VulnCheck, and Test Release runs for the exact tagged commit.RELEASE.*tags remain administrator-created and immutable; secret scanning, push protection, Dependabot alerts, and security updates remain enabled. - Verified before and after tagging. On exact source
a2ef95c035d9ae7cc01469a63926900f1786f9e2, tidy/verify, dependency floors, unit tests, the full race suite, lint, actionlint, source-awaregovulncheck, and nine standard cross-builds passed. Claude Fable 5.1 at xhigh independently returned GO with no blocking findings after a clean-cache Go proxy/sumdb build, 172 functional checks against a temporary SILO server, a 47-command black-box comparison with the published 20260901 binary, and seven additional cross targets. All four exact-SHA main workflows and both publication workflows passed. The immutable public release contains 19 assets; archives and sidecars were checksum-verified, all 11 attested subjects were bound to the tag and source commit, both RPMs were GPG-verified, and thepgsty/mcrelease andlatestimages were independently confirmed as matching amd64/arm64 manifests. The source scan reports no reachable vulnerability.
Compatibility
The full --help tree of this release was diffed against the 20260806 binary: the only difference is the new checksum command. Side by side on the same server, head, cat, get, cp, mirror, find, share, sql, stat, and ls produce identical output and JSON field sets; twenty real-world policy shapes are accepted or rejected identically except for bare ARNs; admin policy info --json is byte-identical. The mc command name and the mcli package name; the ~/.mc / ~/.mcli configuration directories; the github.com/minio/mc module path and all import paths; MC_* environment variables; protocol headers (x-minio-*) and the minio-go user-agent prefix; the minio tier type; the .part.minio transfer suffix; the minio-job scrape job name; and the package formats, asset naming, and YYYYMMDDHHMMSS.0.0 version scheme are unchanged. The client remains compatible with MinIO servers and other S3-compatible endpoints. Projects that consume this module as a Go dependency now resolve github.com/pgsty/silo-pkg/v3 directly and need no replace of their own for it.
Known issues and boundaries
checksum verifyreportsUNKNOWN_READ_ERROR, notMISMATCH, when the server’s own integrity checks refuse a read — SILO’s bitrot protection rejects a corrupted shard before the object API returns it. AMISMATCHmeans the stored checksum describes different bytes than the object API returns, which is the historical write-path defect the command exists to find; anUNKNOWNis never reported as aMATCH.- Access key IDs are redacted wherever they appear inside a credential structure, but a server that reflects a bare access key into an error message will still have it printed: it doubles as the user identifier in legitimate messages. Secrets shorter than three characters are not registered. When the secret key equals the access key — the default
minioadmin:minioadmin— the identifier is redacted from error text too, because the string is the secret. --jsonerror documents list the keys ofcause.erroralphabetically rather than in struct order; the keys and values are unchanged.mcli sqlreports a failed query on stderr but still exits0(pgsty/mc#25); per-object errors are reported without failing the run so a recursive query can continue. Unchanged from upstream.govulnchecktraces GO-2026-5932 in the indirect dependencyx/crypto/openpgp; no fixed version exists and the client does not call it.- JWX 3.2 enforces JWT
critprocessing more strictly, as required by the standard. The terminal-width refresh can also adjust alignment for Indic or ZWJ-heavy text; neither change affected the tested command surface. - minio/mc#5139 continues to apply:
mirror --remove --watchcan delete a live object from the target when a non-current version of it is removed from the source. - Quality debt that does not block the release — archive reproducibility, test-tool portability, deferred coverage of KMS / SSE-S3 / multi-site replication — is tracked in pgsty/mc#21.
Related Commits
- 98a8b0f: build: prepare the 20260903 dependency release
- 6686b64: chore(deps): align the SILO Go dependency stack
- 423490d: fix: stop JSON metrics from panicking
- 14b05e6: build: generate and package third-party notices
- e938cea: fix: align RPM payload signing validation
- 8158605: release: adopt Silo Go SDK v7.3.1
- d6ba236: ci: allow manual release workflow runs
- b6b2655: ci: release an existing tag on dispatch
- 70a2950: fix: preserve embeddable pkg compatibility
- aa07b1c: feat: add read-only object checksum verification
- 1b39423: fix: preserve checksum output in pipelines
- a31a42b: docs: record checksum verify output contract
- 2121081: fix: validate policy writes strictly
- 712bf3a: fix: use regular PUT for empty pipe input
- 9a3805d: ci: make release retries tag-idempotent
- 7bc9843: test: make report permissions portable
- 2d5abc2: chore: complete the AGPL notice on fork-authored files
- f5c5dc9: build: import the SILO package under its own module path
- a8e7eb4: ci: test main on push and stop dispatch runs cancelling each other
- b75d28b: fix: stop checksum verify help from advertising the play alias
- c05da7b: chore: drop configuration for bots that are not installed
- a29b455: fix: stop –debug from printing credentials
- 08244df: fix: keep secrets out of error messages and traces
- 30a7ea0: fix: stop writing support artifacts world-readable
- 14bd724: test: fail the full suite when it panics
- 4abbb3a: feat: expose how much checksum verify actually verified
- f484d0d: ci: make a release prove its commit, and pin what it builds from
- 2c33a4e: chore: ignore the local consumer-module review scratch directory
- bdc742a: fix: close the remaining –debug credential leaks
- 3998392: fix: withhold SSE-C keys and credential URLs the parser rejects
- c1358c4: fix: force 0600 on rotated support artifacts too
- 45e5144: fix: fail –fail-on no-checksum when nothing was verified
- 3674933: ci: make the release gate provable and its claims true
- 8b7b2e9: test: exercise checksum verify against a real server
- 4b36b0a: fix: make credential redaction fail-closed and scrub final output
- a6c14da: ci: verify build provenance before repackaging archives into images
- 915cfd9: fix: reject a malformed app-level global flag instead of ignoring it
- 81e2767: fix: rebuild SigV4 traces from known fields and widen secret scrubbing
- 625713c: fix: keep only the scheme of a signed header and scrub JSON structurally
- ec12b64: fix: redact admin trace output and register every credential ingress
- 1445e88: ci: release only from a signed tag at the tip of main, verify exact provenance
- f188939: fix: close final credential and release verification gaps
- f5b88dd: fix: redact the default admin trace and stop the shape sweep mangling prose
- 59989f8: fix: let svcacct set clear a policy again, report bad global flags once
- d38412a: fix: classify an SSE-C refusal with a key as a read error, keep policy and size messages clear
- b291b53: fix: register a URL password as written, keep listing parameters visible in traces
- 9dfca8f: test: drive the SSE-C refusal classification through the verify path
- 76b2e89: fix: keep an unquoted DSN password whole and match auth schemes case-insensitively
- 328efee: fix: register the password before the first separator of an unquoted DSN payload too
- 7057705: fix: avoid double-closing S3 Select responses
2.16 - mcli 20260806 Released
Published: 2026-08-06 · Version: RELEASE.2026-08-06T00-00-00Z
Two days after mcli 20260804, this release completes the client’s transition to the Silo identity. It is deliberately a pure rebranding and lockdown release: --version and --help now present the Silo client, every remaining path to MinIO’s SUBNET service is disabled at build time, the embedded vendor encryption key is removed from the diagnostics tooling, and the contribution policy moves to no-CLA with a mandatory DCO sign-off. There are no dependency changes and no protocol changes in this cycle — go.mod is byte-for-byte identical to 20260804 — so the regression surface is confined to text, command gating, and CI.
Behavior changes
Every path that previously reached MinIO SUBNET is now disabled at build time and cannot be re-enabled at runtime:
mcli license register,mcli support upload,mcli support proxy set,mcli support callhome enable, and the online-renewal form ofmcli license update ALIASprint a stable notice — “MinIO SUBNET services (registration, licensing, uploads) are disabled in this Silo build of mc; diagnostics remain available locally.” — and always exit1. Drop these calls from scripts. The file-basedmcli license update ALIAS license.keystill works, with the license parsed offline against the bundled public key.mcli support diag/perf/profile/inspectalways operate in local (airgap) mode: reports, profiles, and inspect archives are written to local files and nothing is uploaded anywhere. The--airgapflag is still accepted for compatibility and is effectively always on. SUBNET registration is no longer a prerequisite for any of them.mcli support callhome disable|status,mcli support proxy show|remove,mcli license info, andmcli license unregisterkeep working — they only read or clear local and server-side configuration.- Fresh configurations no longer seed the
playalias pointing at MinIO’s public demo cluster; the defaults are nowlocal,s3, andgcs. Existing configuration files are never modified, and legacy-config migration still recognizes the historical entries. mcli --versiongains an identity line (“Silo object storage client, based on MinIO technology”) and a second copyright line. The first line’s machine-readable format is unchanged, so scripts parsing it are unaffected.
Major Changes
- Silo identity across the CLI: the client introduces itself as “Silo client for object storage and filesystems”. Roughly 220 help texts were reworked: usage lines that refer to the managed server now say “Silo/MinIO server”, example aliases moved from
myminio/playtomysilo, example LDAP DNs moved todc=example,dc=com, and example tier names toSILOTIER-*. Factual references stay factual: theminiotier type, protocol headers, and third-party interop mentions are untouched. - SUBNET disabled at build time: connectivity is compiled out behind a single guard, and the one HTTP choke point that every SUBNET request funnels through refuses with the stable error above. Command entry points gate early, diagnostics force local mode, and the AGPL license notice shown by
mcli license infono longer carries a commercial-subscription pitch. A dedicated regression suite (cmd/subnet-disabled_test.go) pins all of this, so an upstream merge cannot silently reconnect anything. - Governance — no CLA, DCO required: contributions are accepted inbound=outbound under AGPL-3.0-or-later; contributors keep their copyright, and the maintainers collect no rights beyond the project license. Every commit must carry a
Signed-off-bytrailer, enforced by a new CI workflow that matches the trailer against the commit author’s email and exempts only GitHub-issued bot addresses.CONTRIBUTING.md, the PR template, and both READMEs document the policy, and the code-of-conduct contact now points at the fork’s maintainer. - Dual copyright attribution: runtime output and help now credit both lineages —
Copyright (c) 2015-2025 MinIO, Inc.andCopyright (c) 2025-2026 PGSTY— with source builds computing the end year dynamically.NOTICEstates the fork relationship, and the non-affiliation with MinIO, Inc., explicitly. - Release line renamed to
main: workflow branch filters, documentation, and contributor instructions now targetmain; the legacymasterreferences are gone.
Hardening
- Vendor encryption key removed:
mcli support inspectused to fall back to encrypting its output with an embedded MinIO RSA public key whenever no key was supplied — producing archives only the vendor could decrypt. The embedded key is gone: inspect now relies on the server-generated per-request key that is printed to the caller (or an operator-supplied key), and any encrypted-upload path with no configured recipient fails closed instead of silently borrowing a third-party key. Diagnostic output an operator produces is now always decryptable by that operator. - Brand-policy gate:
buildscripts/check-branding.shruns inmake verifiersand in CI. It fails the build if MinIO-operated endpoints, commercial upsell URLs, the upstream product identity, or any embeddedMII…public key reappear in the command tree — while explicitly allowlisting the preserved compatibility identifiers (environment variables, protocol headers, module path, legacy-migration defaults, and original copyright headers).
Engineering and Delivery
- CI moved to the Node 24 Actions line:
actions/checkoutv7,actions/setup-gov7,goreleaser-actionv7, and the Docker action family — all still pinned to commit SHAs, with dependabot keeping the pins current. - Functional tests run against controlled servers only: the suite defaults to a local server (
localhost:9000) instead of MinIO’s public demo cluster, and CI downloads the pinned SILO server releaseRELEASE.2026-08-04T00-00-00Zfrompgsty/silo, verified by SHA-256, before running the suite. - Zero dependency changes: no module updates this cycle; the 20260804 security baseline (Go 1.26.5, zero known reachable vulnerabilities) carries over unchanged.
- Audited before tagging: the release was gated by an independent adversarial review — a full read of the 245-file diff, brand/compatibility grep sweeps, call-graph verification that no command or flag combination can reach
subnet.min.io/play.min.io/dl.min.io, and smoke tests confirming every disabled path returns its stable error with exit status1.
Compatibility
Everything scripts and integrations depend on is deliberately unchanged: the mc command name and the mcli package/binary name; the ~/.mc / ~/.mcli configuration directories (derived from the invoked name); the github.com/minio/mc module path and all import paths; MC_* environment variables; protocol headers (x-minio-*) and the minio-go SDK user-agent prefix; the minio tier type; the .part.minio transfer suffix; the minio-job Prometheus scrape job name; and the package formats, asset naming, and YYYYMMDDHHMMSS.0.0 version scheme. The client remains fully compatible with MinIO servers and other S3-compatible endpoints.
Known issues
The mcli watch regression flagged in the 20260804 notes is resolved on the server side: the fix shipped in SILO 20260804, and this client’s CI now runs the functional suite — including watch — against exactly that release. Pair mcli with SILO server 20260804 or newer to receive bucket events; older published servers remain affected.
Unfixed upstream defects continue to apply, most seriously minio/mc#5139: mirror --remove --watch can delete a live object from the target when a non-current version of it is removed from the source. Exercise caution combining --remove --watch on versioned buckets.
Related Commits
- 8a883ca: ci: move the branch filters to main and fetch the server from pgsty/silo
- 8c304dd: ci: move the pinned actions onto the Node 24 runtime
- 02b1c11: docs: name the release line main, not master
- 810bbd2: ci: pin the functional-test server to a release whose watch API works
- d145647: fix: disable SUBNET connectivity and licensing upsell paths
- 5061c4f: rebrand: adopt Silo identity in CLI help and examples
- c7f7706: docs: align governance files and package metadata with the fork
- 65c71b2: test: default functional tests to a local server and add brand gate
- c62a64d: fix: credit both MinIO and PGSTY in copyright notices
- d205f88: docs: adopt no-CLA plus DCO contribution policy
- 95326ce: docs: add related-projects table and polish contribution wording
- d2c0db7: fix: remove the vendor encryption key and close the proxy-set path
- 0c6704d: fix: repair a link and help text damaged by the brand sweep
2.17 - Silo 20260806 Released
Version: RELEASE.2026-08-06T00-00-00Z · Commit: 3be10fcc1a44f6620ded0bd303461f9d688cca23
SILO 20260806 is the first release published under the Silo name. The previous release, 20260804, was the last one delivered as pgsty/minio; this release completes the cutover to github.com/pgsty/silo and renames every delivery surface — binary, packages, container images, systemd unit, Helm chart — while deliberately preserving every wire and configuration surface a MinIO deployment depends on. On top of the rename it adds native health checking (silo healthcheck), a single-binary distroless container image pilot, complete license-compliance materials in every artifact, and a release pipeline gated on compatibility snapshots and build provenance.
The release covers 28 commits after RELEASE.2026-08-04T00-00-00Z, changing 396 files with 27,188 insertions and 19,561 deletions. It passed a six-phase pre-release acceptance, including a real four-node TLS cluster migration from MinIO to Silo — with byte-verified data integrity, maintenance-gated rolling restarts, fault injection, and a full rollback rehearsal.
Highlights
- The rebrand is complete, and compatibility is the contract. Repository, binary (
/usr/bin/silo), packages (silorpm/deb/apk), images (docker.io/pgsty/silo), and service (silo.service) are renamed; the S3 and admin APIs,/minio/*routes,MINIO_*environment variables,x-minio-*headers, on-disk.minio.sysformat, and Go module paths are all preserved and frozen by a CI compatibility guard. - Native health checking:
silo healthcheck [live|ready|cluster|cluster-read]probes the server’s own health API with correct exit codes, decoded quorum diagnostics, TLS auto-detection, and a--maintenancepre-drain gate — no shell,curl, ormcrequired in the container. - Distroless image pilot:
pgsty/silo:<RELEASE>-distrolessships exactly one program — thesilobinary — ongcr.io/distroless/static, with an exec-formHEALTHCHECKbaked in and/datacreated writable in the image layer. - The classic image does not change behavior: same entrypoint, same bundled tools,
mc ready localkeeps working, and noHEALTHCHECKwas added to it. It now bundlesmcli20260806. - Compliance completed: LICENSE and NOTICE ship in every package and image, CREDITS is regenerated from the actually-linked module set (291 modules) and guarded in CI, and the project adopts a no-CLA, DCO-based contribution policy.
- Components refreshed: embedded SILO Console 2.1.1,
silo-pkg3.11.0,mcli20260806, Go 1.26.5. - Provenance-gated releases: container images are built only from published, checksum- and attestation-verified release archives; image SBOMs and provenance attestations now cover the distroless variant too.
The rename
What changed, and what deliberately did not:
| Renamed (delivery surface) | Preserved (compatibility surface) |
|---|---|
Repository: github.com/pgsty/silo (main branch) |
S3 API, admin API, and request signing behavior |
Binary: /usr/bin/silo |
/minio/* routes, including /minio/health/* and metrics |
Packages: silo-*.rpm, silo_*.deb, silo_*.apk |
MINIO_* environment variables and x-minio-* headers |
Images: docker.io/pgsty/silo (+ -distroless) |
On-disk format (.minio.sys), erasure coding, versioning |
Unit: silo.service (conflicts with, and supersedes, minio.service) |
Go module and import paths (github.com/minio/...) |
Default config dir: ~/.silo (falls back to an existing ~/.minio) |
mc compatibility alias for the bundled mcli |
The server presents its own identity — silo --version reports the AGPL-3.0 license, MinIO’s 2015-2025 copyright, PGSTY’s modification copyright, and the “based on MinIO technology” attribution — and every inherited connection to MinIO-operated services (the update feed and its signing key, SUBNET, telemetry) is severed rather than redirected. The container entrypoint translates the legacy minio argv token, so docker run pgsty/silo minio server /data keeps working.
A snapshot-based rebrand guard runs in CI: it fails on any drift, in either direction, across 334 route literals, 437 environment tokens, 84 headers, and 9,014 exported symbols.
Native health checking
The server binary can now probe its own health endpoints, which makes container health checks possible without any second binary — and is what the distroless image relies on:
- The check vocabulary maps 1:1 onto
/minio/health/<path>;live(the default) answers “is this process serving,”readyadds KMS/etcd reachability when configured, and theclusterpair evaluates write/read quorum across every erasure set. - Exit codes are
0(healthy) and1(anything else) — never the Docker-reserved2. One diagnostic line decodes the server’sx-minio-server-statusand quorum headers fordocker inspect;--jsonemits a machine-readable verdict. - The probe target is derived the way the server derives its own listen address:
--address/MINIO_ADDRESS, with HTTPS auto-detected frompublic.crt+private.keyin the certs directory, or overridden wholesale with--url/MINIO_HEALTHCHECK_URL. The environment form exists because a probe process cannot see the server’s command line — if the server’s address or TLS comes from CLI arguments, one environment variable redirects the baked-in probe. silo healthcheck --maintenance clusteranswers the pre-drain question: exit0means the node can be taken down without losing HA; HTTP 412 (exit1) means it cannot.- Certificate verification is skipped, matching the kubelet’s documented behavior for HTTPS probes, and the transport ignores
HTTP_PROXYso loopback probes never route through a proxy.
Kubernetes needs none of this — kubelet httpGet probes hit /minio/health/live and /minio/health/ready from outside the container — and the cluster checks should stay out of per-container probes: they reflect cluster-wide quorum, not one process. The full design rationale, including verified endpoint semantics, is recorded in the health-check design note.
Distroless image pilot
Alongside the classic image, this release publishes a distroless variant: pgsty/silo:RELEASE.2026-08-06T00-00-00Z-distroless, plus a rolling distroless tag.
- Base is
gcr.io/distroless/static-debian12: CA certificates, tzdata,/tmp, and an/etc/passwdwith anonroot(65532) entry — no shell, no package manager, no libc. On top of it, exactly one program:/usr/bin/silo(plus the license set under/licenses/). The image is 128 MB versus the classic 199 MB. - The binary is the
ENTRYPOINT; an exec-formHEALTHCHECKrunningsilo healthcheck readyis baked in (interval 30s, timeout 10s, start-period 2m, retries 3), so Compose users get workingdepends_on: condition: service_healthywith zero configuration. /datais created in the image layer, world-writable — there is no entrypoint left to repair volume ownership at runtime, and this is what makes every privilege mode work,--userincluded. This fixes, for the distroless variant, the non-root failure documented in #55.- Not supported in this variant: the deprecated
MINIO_USERNAME/MINIO_GROUPNAMEprivilege-drop path (use--useror KubernetesrunAsUser),docker exec <c> shdebugging (use ephemeral-container tooling), and in-imagemc(use the releasedmclior the client image). - TLS: mount certificates at
/tmp/.silo/certs(the container’s default certs directory) and both the server and the baked-in probe derive HTTPS from the same location; for CLI-configured servers, setMINIO_HEALTHCHECK_URL.
The classic image remains the default and is unchanged. If the pilot proves out, the distroless variant becomes the recommended image later; the decision record lives in the design note above.
Container images
The classic image was diffed field by field against pgsty/minio:RELEASE.2026-08-04T00-00-00Z: entrypoint, exposed ports, volumes, working directory, user, and (absent) health-check configuration are identical. Exactly three differences exist, all deliberate: Cmd is ["silo"] instead of ["minio"], the upstream update-verification key variable MINIO_UPDATE_MINISIGN_PUBKEY is removed (updates through upstream channels are permanently disabled), and HOME=/tmp is declared to match the entrypoint’s writable-home guarantee.
The bundled client is upgraded to mcli RELEASE.2026-08-06T00-00-00Z (with the mc alias preserved), pinned by per-architecture SHA-256 digests and verified against the published checksums at build time. Interoperability of the released mcli 20260806 against this server — multipart, versioning, presigned URLs, metadata/tags, user and policy administration — was verified as part of release acceptance.
Helm chart
The chart ships as silo 7.0.1, preserving rendered resource identity with the legacy chart across a simulated upgrade (verified by the migration guard over 7 rendered resources). Its default image tag now points at this release — docker.io/pgsty/silo is a fresh repository, so the inherited default could never have pulled. The chart still ships no liveness/readiness/startup probes; adding them is planned, and documented, in the design note’s follow-up phase.
Packaging and migration
RPM, DEB, and APK packages install exactly six files: /usr/bin/silo, silo.service, a sysusers definition (creating the silo system user), /etc/default/silo (marked config/noreplace), LICENSE, and NOTICE. RPMs are GPG-signed with the PGSTY maintainer key (9592A7BC 7A682E73 33376E09 E7935D8D B9BD8B20). RPM and DEB now carry a unified, PGDG-style 1PGSTY release segment — silo-<version>-1PGSTY.<arch>.rpm and silo_<version>-1PGSTY_<arch>.deb — replacing the inherited bare -1 on RPM and the missing revision on DEB; APK names stay bare because Alpine pkgrel admits only -r<integer>.
silo.service is designed for takeover: Type=notify (readiness is signaled by the server itself), Conflicts=minio.service + After=minio.service (starting Silo stops a running MinIO unit), and two environment files — /etc/default/minio is read first and /etc/default/silo overrides it — so an existing MinIO configuration is inherited without editing. For existing deployments whose data is owned by the minio user, the documented drop-in keeps ownership untouched:
Distributed migrations must switch all nodes together.
Cluster bootstrap verifies that every node runs the same binary (by checksum). A mixed cluster — some nodes on Silo, some still on MinIO — does not form: the new node stays in activating, logging Expected Silo binary checksum ... seen: ... and Waiting for at least 1 remote servers with valid configuration, indefinitely. Stop MinIO on all nodes, then start Silo on all nodes (near-simultaneously). Once every node runs Silo, rolling restarts work normally — gate each one with silo healthcheck --maintenance cluster.
Migration troubleshooting, from the acceptance run: if Silo starts as the packaged silo user against a deployment whose TLS certificates live under the minio user’s home, it fails with HTTPS specified in endpoints, but no TLS certificate is found and restart-loops until the systemd start limit — the legacy-user drop-in above is the fix. Keep the MinIO package and unit installed (disabled) during the migration window: the rollback path — stop Silo, start MinIO — was rehearsed and reads all data written during the Silo window, because the migration touches neither data ownership nor format.
Components and dependencies
- SILO Console 2.1.1 — the embedded console, selected from
pgsty/silo-consolewhile preserving thegithub.com/minio/consoleimport path. silo-pkg3.11.0 — retains the policy/LDAP/certificate fixes including the LDAP-over-TLS repair tracked in #15.mcli20260806 — bundled in the image and released separately; see its release notes.- Go 1.26.5 — toolchain unchanged from 20260804.
Build, CI, and release pipeline
- Compatibility as a CI gate: the rebrand guard snapshots routes, environment tokens, headers, metrics, storage/policy identifiers, and exported symbols, and fails on any unreviewed drift; companion scripts assert the delivery surface (binary path, unit contents, image layout) and that no live upstream endpoint remains in runtime code.
- Release-image gate: every release-pipeline run builds both container images and asserts, among others: the distroless
HEALTHCHECKsurvives into the image config (it is a Docker extension outside the OCI spec),/dataships world-writable, no shell and no/usr/bin/minioexist, Docker’s health state turns healthy from the baked probe alone, and SIGTERM still stops the server gracefully as root and as--user 1001:1001. - Provenance chain: images are built from the published release archives after checksum verification and
gh attestation verifyagainst the exact tag; per-architecture SBOMs and provenance attestations are pushed for the classic and distroless images; the distroless health-check gate runs before the multi-arch manifests are promoted. - Workflow runtime moved to Node 24 across CI actions.
Compatibility and upgrade notes
- Package upgrades are a takeover, not an in-place update. Install
silo, keep/etc/default/minioas is (it is inherited), enablesilo.service; starting it stopsminio.servicevia the conflict relation. Data is untouched. - Keep data ownership stable with the legacy-user drop-in above; do not chown storage or move certificates during migration.
- Distributed clusters: full-stop switchover only. See the warning above — mixed Silo/MinIO nodes do not form a cluster.
- Container users: the image is now
docker.io/pgsty/silo;docker.io/pgsty/miniostays frozen at 20260804 as an archive. The classic image’s behavior is unchanged — includingmc ready localhealth checks — and the distroless variant is strictly opt-in. - Distroless differences are deliberate: no shell, no in-image
mc, noMINIO_USERNAMEpath; health is native; servers configured via CLI arguments needMINIO_HEALTHCHECK_URLfor the baked-in probe. - Helm users: chart 7.0.1’s defaults now pull this release; override
image.tagexplicitly if you pin versions. - Known and unchanged: the classic image still does not create
/datain the layer, so fully non-rootdocker runagainst a Docker-managed volume fails as before (#55, fixed in the distroless variant); the inherited Postgres/MySQL legacy notification-migration limitation from the 20260804 notes still applies (#53). - Pair with
mcli20260806 for the client side; older clients continue to work over the unchanged wire protocol.
Verification
This release was verified in stages, each with recorded evidence:
- unit and end-to-end matrices for the health-check command: target derivation and precedence (flag/env/derived), real-TLS auto-detection, exit-code contract, JSON schema, timeout bounds, usage errors;
- cluster-semantics verification on a four-node cluster: with 2 of 4 nodes stopped,
clusterreports 503 withwrite-quorum=5whilecluster-readandlivestay 200 — the write/read quorum split observed live, matching the erasure math; - image acceptance: the classic image diffed field-by-field against the 20260804 baseline; the distroless image asserted down to file inventory, exact health-check configuration, and root/non-root/TLS/env-override runtime scenarios;
- an adversarial model-based code review of the new code, with every confirmed finding fixed and re-verified;
- a six-phase pre-release acceptance concluding in a real migration: a Pigsty-deployed four-node TLS MinIO 20260804 cluster (16 drives, EC:4) was migrated to Silo via the packaged takeover path — reference data (multipart, versioned, tagged objects) read back byte-identical, four maintenance-gated rolling restarts,
kill -9fault injection with the load balancer serving 23/24 continuous IO rounds (the only failure in the kill second), Prometheus metrics continuity, and a full rollback to MinIO and back, proving the migration reversible.
Validation boundaries
Not proven by this release and not to be inferred: external LDAP/OIDC/KMS/etcd services (the only case where ready diverges from live was not exercised against a live KMS); amd64 packages were cross-built and payload-checked but not installed on a physical x86-64 host; the renamed Docker publish workflow (including the new SBOM/attestation lanes) has its first production run at this release’s publication; Windows and Intel macOS were not tested.
Artifacts
- GitHub release
RELEASE.2026-08-06T00-00-00Zatpgsty/silo, with checksummed platform archives, provenance attestations, and RPM/DEB/APK packages (GPG-signed RPMs); docker.io/pgsty/silo:RELEASE.2026-08-06T00-00-00Zandlatest;docker.io/pgsty/silo:RELEASE.2026-08-06T00-00-00Z-distrolessanddistroless— published on demand from the finished release;- companion releases:
mcli20260806,silo-pkg3.11.0, embedded SILO Console 2.1.1; - design record: Native Health Checks and the Distroless Image.
Selected changes
15def34dc,77bdc4c0c: drop upstream delivery residue; present Silo identity and close inherited upstream services15ab10833: rename the delivery artifacts to silo and complete the package payload30749911b: ship the silo binary in the image and translate the legacy argv commande071bb77e: replace the minio chart with a silo chart that preserves identitybd8df5166: gate the rebrand on compatibility, packaging, and provenance evidence6613c2a3c: pin the external test fixtures and run the suites against the silo binaryfd2ca1c6d,c46b16ec6,c47733abc,f1c77d5a2: cut over to pgsty/silo and main; document the archived branch6740e6978: move the workflow actions onto the Node 24 runtimeb57275be3: adopt the no-CLA plus DCO policy and fix copyright terms62717d7bf,a6d6d9b02: update the embedded Console to 2.1.0, then 2.1.16bd9cf77e: regenerate CREDITS from the linked module set and guard it in CI219670d31: ship LICENSE and NOTICE in every package and image2ff594f4b: add the nativesilo healthchecksubcommand4c34d2309: add the distroless image variant as a pilotb6d47b739,9462cce16: harden both per adversarial review; lint cleanup16b78eb4e: bundle mcli 20260806 and point the Helm defaults at this release062a91bee: pin the CREDITS module closure to the shipped linux target467931455: unify the rpm and deb release segment as1PGSTYb14ea22aa: match checksum manifest entries exactly in the image publish lane3be10fcc1: add a manual finalize lane refreshing SBOMs and checksums for signed Draft packages
Acknowledgments
Four contributors have code merged into this fork, and the Git history carries their authorship: @ZouhairCharef patched CVE-2026-34986 in go-jose (#18), @mfredenhagen patched CVE-2026-39883 in OpenTelemetry (#19), @pinginfo implemented Flush on trackingResponseWriter to repair bucket notification streaming (#34), and @waterkip repointed the documentation links to the Silo portal (#41).
A first release under a new name is also the right moment to thank everyone who has filed issues against this fork — bug reports, compatibility findings, and proposals alike, resolved and still open:
@mosesdd (#1), @Xavier-777 (#2, #17), @jiadzh (#3), @TLINDEN (#4), @AntonOfTheWoods (#5), @zylpsrs (#6), @nsanitate (#7), @makinikm (#9), @magicxor (#10), @spaceg00se-r (#11, #14), @heroes1412 (#13), @vampywiz17 (#15), @davinkevin (#20), @chalukyaj (#30), @cbornet (#31, #32), @jvasile (#33), @Kesavaambati (#35), @redfoxfox (#38), @kuldeep-link11 (#39, #40), @meesudzu (#42), @pmezhuev (#43), and @kh0mka (#51).
Several of this release’s headline items trace directly back to those reports: the bundled-client guarantee to #4 and #9, the LDAP-over-TLS repair to #15, the completed package payload to #33, GPG-signed RPMs to #43, the migration guide to #42, and the distroless /data fix to #55.
Pull requests still in flight deserve a mention too. @davinkevin’s distroless image PR (#21) anticipated this release’s pilot months in advance — the shipped variant supersedes that PR with the native health check built in, but the direction was proposed there first. Conformance PRs from @magicxor (#12) and @ycjlin (#37) are queued for review immediately after this release.
Everyone who has contributed to this fork is recorded in CONTRIBUTORS.md, which is now the project’s attribution record — GitHub generates no contributor graph for forks.
2.18 - Silo 20260804 Released
Version: RELEASE.2026-08-04T00-00-00Z · Commit: d88f46ccee345a9c2fabe2d221d9a9e56bc11aec
SILO 20260804 is a security, correctness, and release-engineering update to the pgsty/minio community fork. It completes the internode storage-containment work begun with CVE-2026-42600, prevents request-controlled values from impersonating server-calculated S3/IAM policy conditions, restores streaming flush behavior, fixes several multipart and versioning edge cases, hardens notification configuration migration, moves the build baseline to Go 1.26.5, and connects the server to the SILO-maintained Console, shared package, and mcli releases. The release pipeline was rebuilt to produce reproducible binaries and GPG-signed packages.
The release covers 50 commits after the pre-2026-06-18 baseline, changing 155 files with 9,241 insertions and 981 deletions. Every change was reviewed against the tagged commit and verified on macOS ARM64 and Linux AMD64, with GitHub CI green on the released HEAD.
Highlights
- Internode containment completed: validates storage-REST message bodies, storage Grid frames, and peer-S3 Grid requests at the storage boundary, closing the remaining path, volume, erasure-metadata, panic, and unbounded-allocation defects left after removing
ReadMultiple. - S3/IAM decisions now use effective values: client input can no longer shadow internal condition values; request tags and existing-object tags are separated;
s3:signatureAgeis confined to verified presigned requests; ands3:versionidfollows the version the server actually acts on. - Bucket and object resources are separated: twelve sensitive bucket-level writes are no longer authorized through an object-only
bucket/*resource pattern. A documented compatibility switch is available for migration. - Multipart compatibility and correctness improved: full-object checksum completion works without per-part checksums when the protocol permits it, zero-length multipart checksums are preserved, and duplicate part numbers are rejected instead of assembling duplicated data.
- Streaming reliability restored:
trackingResponseWriternow implementsFlushcorrectly and records implicit HTTP 200 responses, repairingmcli watch, bucket-notification listeners, and S3 Select keep-alives affected by the inherited regression documented in the 20260618 release. - Notification configuration hardened: NATS and AMQP keys used by parsers and legacy migration are registered and round-trip correctly; libpq connection parameters are quoted safely; invalid-key errors no longer echo secret values.
- Reproducible, signed release pipeline: binaries no longer embed the build machine’s paths, packages install under the canonical systemd path, and RPMs are GPG-signed. The container entrypoint now shuts down gracefully on every privilege path.
- Release baseline refreshed: Go 1.26.5,
klauspost/compress1.18.7, Apache Thrift 0.24.0, SILO Console 2.0.0,silo-pkg3.11.0, andmcli20260804.
Security Hardening
Internode storage and Grid containment — SN-2026-002
Removing the obsolete ReadMultiple endpoint in 20260618 closed one reachable path but did not close the underlying defect class. Storage-REST request bodies and Grid RPC frames do not pass through the HTTP query-validation middleware, and peer-S3 RPCs can bypass the storage-REST wrapper entirely.
This release moves containment to the storage boundary and validates every caller-controlled path, volume, erasure parameter, part size, shard length, and allocation length before use. The fixes include:
- reject traversal on both path and volume axes, including Windows volume-root aliases;
- cover peer-S3 bucket RPCs that reach drives without the storage-REST wrapper;
- reject zero or unusable data/parity/block-size combinations before shard arithmetic;
- reject negative part sizes and truncated shards instead of reporting them healthy;
- cap storage-REST
ReadFileallocations at 5 GiB; - bound other allocations derived from internode declarations;
- contain panics in deadline-bounded storage work without blocking the caller;
- preserve
ReadPartserrors across keep-alive responses and avoid the empty-part trace panic.
These routes require cluster-root or internode credentials and are registered only in distributed-erasure deployments. Single-node S3 behavior is unchanged. See Internode Path Containment Audit for the protocol-surface analysis.
Effective policy-condition values — SN-2026-003
The policy condition map historically mixed values calculated by the server with raw request entries. A client-controlled spelling could therefore shadow or synthesize an internal condition value. SILO 20260804 pairs silo-pkg 3.11.0’s exact-key lookup rule with server-side source normalization:
- internal condition names cannot be supplied as arbitrary client values;
s3:prefix,s3:delimiter, ands3:max-keyscome from their effective query inputs;- header-backed
x-amz-*conditions do not accept unrelated query substitutes; - when storage class or upload tagging supports both forms, an explicitly present Header wins, including an empty Header;
s3:ExistingObjectTag/*comes only from stored object metadata;s3:RequestObjectTag/*is bound to the tag input consumed by the relevant operation;s3:signatureAgeis exposed only after verified SigV4 presigned authentication calculates it;s3:versionidis absent when no version is named and is rebound perDeleteObjectsentry to the effective resolved version.
The version-ID behavior closes the fail-open trap that a superficial “omit empty values” fix would have created for Multi-Delete. See Absent Is Not Empty.
Bucket/object resource boundary — SN-2026-004
The IAM matcher used to append a slash to a bucket-level request, allowing an object-only resource such as arn:aws:s3:::bucket/* to authorize selected bucket-level operations. This release withholds twelve sensitive writes from that pattern on Allow statements:
PutBucketPolicy, DeleteBucketPolicy, PutBucketObjectLockConfiguration, PutBucketVersioning, PutReplicationConfiguration, PutBucketLifecycle, DeleteBucket, ForceDeleteBucket, PutBucketCors, DeleteBucketCors, PutBucketQOS, and PutInventoryConfiguration.
Deny and NotResource behavior is unchanged. Read/list operations, CreateBucket, bucket tagging, default encryption, and notification configuration remain compatible. Built-in policies use Resource: "*" and are not affected.
Policy migration required for custom bucket grants
If a custom policy grants one of the twelve actions — often through s3:* — using only arn:aws:s3:::bucket/*, add the bare bucket ARN:
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on restores the historical matcher while policies are migrated. It also restores the historical over-grant, so use it only as a temporary rollback control.
Trusted client-address boundary
MINIO_API_TRUSTED_PROXIES provides an enforceable, opt-in boundary for aws:SourceIp, audit remotehost, event notification Host, and the client address shown by mcli admin trace:
- set it to an address/CIDR list to trust forwarding headers only from those peers and walk forwarding chains from right to left;
- set it to
noneto ignore all forwarding headers; - leave it unset to preserve historical behavior exactly.
The old _MINIO_API_XFF_HEADER=off switch still suppresses only X-Forwarded-For; it does not protect against X-Real-IP or RFC 7239 Forwarded. If IP-based policy is part of your security boundary, configure trusted proxies explicitly and prevent direct access to the S3 API port. Multi-node deployments should allow their own node addresses. See Client Source Address Trust.
S3 and Storage Correctness
Multipart upload
- CompleteMultipartUpload accepts the S3 full-object checksum mode when the completed request supplies no per-part checksums and the upload metadata does not require them.
- The checksum of a zero-length multipart object is retained instead of being discarded as empty metadata.
- Part numbers must be strictly increasing. Duplicate entries such as
[1,1]now returnInvalidPartOrderinstead of consuming the upload and assembling the same part twice. Legal part lists with gaps or a non-1 start remain accepted. See Duplicate Part Numbers.
Object reads and buffer ownership
- erasure reads again pool buffers only where ownership permits reuse;
- update downloads return caller-owned buffers instead of exposing data that can be overwritten after return;
- the old HTTP streaming helpers orphaned by
ReadMultipleremoval are deleted after reference and platform-tag checks.
HTTP response tracking and S3 Select
trackingResponseWriter.Flush()delegates to the underlying flusher and commits the response state correctly;- the first implicit write records HTTP 200, preserving audit and metric accuracy;
- S3 Select tests no longer race a client parser against response-body ownership;
- CSV, JSON, and Parquet selection paths remain covered, including range/error and keep-alive behavior.
The inherited silent-flush regression called out in SILO 20260618 is therefore fixed in this release.
IAM, Versioning, and Audit Behavior
- DeleteObject and each entry in DeleteObjects evaluate
s3:versionidagainst the effective version selected by the server. - Request tags can no longer impersonate existing-object tags during policy evaluation.
- The
merrstag is restored when dangling-object deletion records are emitted, preserving the intended audit classification. - Bucket-policy and IAM paths share the hardened condition-source rules while retaining their established S3 routing and error behavior.
Notification Configuration
- registers the NATS
user_credentials,nkey_seed, andtls_handshake_firstkeys read by the parser; - separates the legacy NATS environment-variable spelling from the stored config key;
- repairs NATS migration round trips and the AMQP
immediate/internalmapping; - adds a mechanical audit that compares keys read and written by notification code with each subsystem’s registered schema;
- quotes libpq connection-string parameters so whitespace, quotes, and backslashes retain their intended value;
- prevents invalid-key diagnostics from echoing secret values.
See Notify Keyspace Registration.
Known legacy migration limitation
The inherited Postgres and MySQL legacy migration functions still write the unregistered host, port, username, password, and database fields. A migrated configuration can therefore fail validation on the next load, and notification target loading is fail-fast across subsystems. This predates 20260804, but upgrades from pre-connection-string database notification configurations must be reviewed and converted before restart. The stored password field may contain a plaintext database password.
Components and Dependencies
- Go 1.26.5: includes security fixes in
crypto/tlsandosplus compiler, runtime, networking, and syscall corrections. klauspost/compress1.18.7: refreshes the compression stack used by object and archive paths.- Apache Thrift 0.24.0: updates the dependency compiled through Parquet support.
- go-systemd 22.6.0: deliberately retained instead of 22.7.0 because the later version introduced a NetBSD clock dependency incompatible with the supported cross-build matrix.
- SILO Console 2.0.0: the embedded console is selected from
pgsty/silo-consolewhile preserving the compatiblegithub.com/minio/consoleimport path. silo-pkg3.11.0: provides the companion policy, LDAP, certificate, RNG, and time-format fixes while preserving thegithub.com/minio/pkg/v3module path.mcli20260804: the embedded client comes frompgsty/mc; release images expose it asmcliand keep themccompatibility alias.
See the companion release notes for silo-pkg 3.11.0, mcli 20260804, and SILO Console 2.0.0.
Build, CI, and Packaging
This release rebuilt the release pipeline for reproducibility and supply-chain integrity:
- Graceful container shutdown on every path. The entrypoint’s custom UID/GID branches now
execinto the server so it runs as PID 1 and receivesSIGTERMdirectly; previously those branches left an intermediate shell as PID 1 and the server was killed at the container stop timeout. A CI smoke test builds the release runtime image and asserts graceful shutdown on both the default and drop-privilege paths. - Reproducible binaries. Release binaries no longer embed the build machine’s
GOPATH/GOROOT, so-trimpathholds and a third party rebuilding the tag gets matching bytes. The published Linux binary contains no build-host path. - Hardened release workflow. The release tag is passed through the environment and whitelisted rather than spliced into the shell, the build is checked out at the tag being released, and an untracked shadow GoReleaser config that could publish or move
latestout of band was removed. - Honest gates. CI gates build, vet, unit tests, lint, generation drift, race tests, and cross-compilation; the cross-compile matrix is aligned to the exact set of published targets; and the lint and dependency-install steps now fail on real errors instead of masking them.
- Signed, canonical packages. RPM, DEB, and APK packages are produced with nFPM under the PGSTY identity, the systemd unit installs at
/usr/lib/systemd/system/minio.servicewithType=notify, and RPMs are GPG-signed offline with the PGSTY maintainer key (fingerprint9592A7BC 7A682E73 33376E09 E7935D8D B9BD8B20). - Release and container publication remain separate gates. GoReleaser produces the platform archives, checksums, and packages; the multi-architecture image is published on demand from the finished release. A local snapshot does not prove a public release or image exists.
Compatibility and Upgrade Notes
- Keep every node on one release during a cluster rollout. Internode validation changed across storage-REST and Grid surfaces; mixed binaries were not production-tested.
- Audit custom IAM policies. Add the bare bucket ARN for the twelve protected bucket writes. Use
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=ononly as a temporary migration control. - Configure client-address trust deliberately. If
aws:SourceIpor audit attribution matters, setMINIO_API_TRUSTED_PROXIESand close direct network paths around the proxy. - Review legacy database notification settings. Convert Postgres/MySQL host/user/password fields to the supported connection-string format before restart.
- Expect duplicate multipart completion entries to fail. Clients sending the same part number more than once now receive
InvalidPartOrderinstead of a corrupted successful object. - Use the matching
mcli. The 20260804 client disables self-update and must be upgraded through packages or GitHub Releases;mcli updateremains as a compatibility command but exits non-zero. - RPM users can enable signature verification. Packages are signed with the maintainer key above; import it before enabling
gpgcheckfor the SILO packages.
Verification
Changes were reviewed against the tagged commit and re-verified rather than trusted from prior reports:
git diff --check, gofmt, module verification, and YAML/shell syntax;go build ./...,go vet ./..., project lint, andgovulncheck ./...;- full
go test ./..., the complete race suite, and repeated race tests over storage, policy, notification, HTTP tracking, and S3 Select changes; - generator idempotence plus deliberate stale-source and untracked-output counterexamples;
- cross-compilation across every published target;
- Linux AMD64 native full tests, targeted race tests, live S3/
mclismoke tests (create/upload/download/copy, range, versioning, delete markers, health checks, graceful shutdown, restart persistence), and systemd notify behavior; - release-artifact verification: GitHub CI green on the released HEAD, reproducible binaries with no build-host path, the systemd unit installed at
/usr/lib/systemd/system, and RPM signatures validated withrpmkeys --checksig.
govulncheck found no vulnerability reachable from the Server or mcli code. One module-level notice remains for the unmaintained golang.org/x/crypto/openpgp package (GO-2026-5932); that package is not imported into these binaries.
Validation boundaries
The following were not proven by this release and must not be inferred from cross-compilation or unit tests:
- native Windows execution and Windows filesystem semantics;
- Intel macOS and physical Linux ARM64 hosts;
- a production multi-node rolling upgrade, site replication, or lifecycle expiration run;
- real reverse-proxy chains and direct-ingress isolation;
- external LDAP, OIDC, KMS, STS, Postgres, MySQL, NATS, and AMQP services;
- installing and upgrading the signed package under a real systemd host.
Artifacts
- GitHub release
RELEASE.2026-08-04T00-00-00Zwith checksummed platform archives for Linux, Darwin, and Windows on amd64 and arm64; - RPM, DEB, and APK packages under the PGSTY identity, with GPG-signed RPMs;
docker.io/pgsty/minio:RELEASE.2026-08-04T00-00-00Zand the release-selectedlatesttag, published on demand from the release;- matching SILO Console 2.0.0,
silo-pkg3.11.0, andmcli20260804 references.
Selected Changes
ca7baa670,80e8eaa42,b6f70ab08: validate internode paths, erasure metadata, and allocation sizesa36fd8fff: contain panics in deadline-bounded storage work2f55347f7: bind S3/IAM policy conditions to effective request values744a9dcd7: binds3:versionidto the effective object version97b7d2804: enforce the bucket/object resource boundaryfe6dc4780: add the trusted-proxy client-address boundary22c1e41fd: reject duplicate multipart part numbersc8590413f,3e14733f1: restore full-object and zero-length multipart checksum behavior8069a32ac,65795ee1f: restore response commit and streaming flush semantics162ded343,0c14d8151: repair notification key registration and libpq quoting924717926,89d346bf5: restore safe buffer pooling and returned-buffer ownership3b8a55dee: exec into the dropped-privilege process so signals reach the server2ca4971d9: stop stamping the build machine’s paths into the binary4c185d5a6,e064b5555: harden the release workflow and remove the shadow configaa5139369: install the systemd unit under/usr/lib11d79fddc,ca674a696,021110b45,d88f46cce: gate build, vet, tests, lint, generation, race, and cross-compilation, and smoke-test the release image
2.19 - mcli 20260804 Released
Published: 2026-08-04 · Version: RELEASE.2026-08-04T00-00-00Z
This is the first release of the pgsty/mc community fork since 20260417. It fixes a credential leak in debug logging, severs every remaining connection between the client and upstream release channels, moves containers and packages onto artifacts this fork builds itself, and migrates packaging from MinIO’s pkger to standard nFPM — with GPG-signed RPMs for the first time.
Upstream minio/mc was archived in July 2026. Its final commit, 77f82e18, is exactly this fork’s base, and upstream never cut a release containing it — so this build is strictly newer than any official mc binary ever published.
Behavior change
mcli update self-update is disabled in this fork. The command remains for script compatibility and still accepts its original arguments, but it no longer contacts the network or replaces its own binary; it prints an explicit notice and always exits with status 1. Upstream mc update exited 0 when already up to date, so drop the call from any script that treats a non-zero exit as failure. Upgrade through the Pigsty package repository or GitHub Releases.
The automatic version check that ran against upstream release feeds on every invocation has also been removed entirely. The MC_UPDATE and MINIO_UPDATE environment variables are no longer consulted.
Major Changes
- Self-update disabled, upstream release channels severed: the
minio/selfupdateandaead.dev/minisigndependencies and all binary-replacement logic are gone, along with the update notifier and the FIPS/non-FIPS update paths. Theupdatecommand survives as a compatibility shell, and the runtime helpers (Docker / DCOS / Kubernetes / source-build detection) moved to a dedicatedcmd/runtime-info.go. The client previously reached out to upstream release feeds on every invocation to print an upgrade hint; there is now no outbound release probing at all. - Containers and artifacts fully localized: the default image is built from the checked-out fork source, and hotfix binaries are copied from the local build context — no upstream prebuilt binaries are downloaded. The upstream publishing files
Dockerfile.release,Dockerfile.release.old_cpu, anddocker-buildx.shwere removed, and the obsolete MinIO hotfix upload target is disabled. - Packaging migrated to nFPM: replaced MinIO’s
pkgerwith standard nFPM. Artifact layout and install path are unchanged (/usr/local/bin/mcli, package namemcli,YYYYMMDDHHMMSS.0.0version scheme), but the vendor is now PGSTY, the license uses the SPDX identifierAGPL-3.0-or-later, and the DebianSectionmoved from empty toutils. - RPMs are now GPG-signed: RPMs are signed offline with the maintainer key (fingerprint
9592A7BC7A682E7333376E09E7935D8DB9BD8B20). All package metadata is asserted before signing, and the signature is re-verified with checksums regenerated afterwards. DEB and APK packages carry no package-level signature; their trust anchor lives at the repository layer. - Build provenance hardened: every previously published binary was stamped by the Go toolchain as built from a modified working tree (
vcs.modified=true), which broke the link between an artifact and its Git tag. This release fixes that and adds an enforcing check to both the release and test pipelines, so every binary is traceable to an exact commit.
Security Fixes
- SUBNET credentials redacted in debug logs: with
--debugenabled, SUBNET HTTP exchanges are printed in full. Previously theapi-key/api_keyquery parameters, authentication headers, and response bodies all reached the log in clear text — and SUBNET’s authentication and registration endpoints return API keys, licenses, and tokens in their responses. Both parameter spellings and duplicate values are now masked uniformly, sensitive response headers are redacted, and SUBNET response bodies are excluded from debug dumps entirely. The leak is inherited from upstream and present in every previous release, upstreammcincluded: if you have ever shared--debugoutput of SUBNET commands (mcli license .../mcli support ...), treat the API keys and licenses in it as exposed and rotate them. - Redaction isolated from caller state: debug tracing now dumps copies of the request and response, so redaction cannot mutate objects the caller still holds. Zero-length, fixed-length, and unknown-length response bodies are all covered, and callers can still read the response normally.
Dependency Updates
This cycle’s dependency work is security maintenance, not routine hygiene: every bump below except the term / mod / sync / tools refresh closes at least one published advisory in the Go vulnerability database, and govulncheck reports zero known vulnerabilities reachable from this release’s code. No security advisory has ever been published for minio/mc, minio-go, madmin-go, or minio/pkg themselves.
- Go build baseline upgraded from
1.26.2to1.26.5(the newest 1.26.x at release time), picking up the 1.26.3–1.26.5 security batches — including GO-2026-4970 (symlink-based root escape inos) and GO-2026-5856 (Encrypted Client Hello privacy leak incrypto/tls), the two most relevant to an S3 client that writes local files and speaks TLS. github.com/klauspost/compressfromv1.18.5tov1.18.7(closes GO-2026-5841).github.com/prometheus/prometheusfromv0.310.0tov0.311.3(closes GO-2026-5264, GO-2026-5381, GO-2026-5710).google.golang.org/grpcfromv1.79.3tov1.82.1(closes GO-2026-6061), with thegenprotofamily refreshed alongside.- The
golang.org/x/*family refreshed across the board:cryptov0.49.0→v0.53.0(the 14-advisory GO-2026-5005…5033 batch),netv0.52.0→v0.56.0(GO-2026-5025…5030 and GO-2026-5942),sysv0.42.0→v0.46.0(GO-2026-5024),textv0.35.0→v0.39.0(GO-2026-5970), plusterm,mod,sync, andtools. - Removed
aead.dev/minisignandgithub.com/minio/selfupdate, and synchronized the third-party credits file.
Engineering and Delivery
- Integration test dependencies pinned: CI no longer downloads the MinIO server from a mutable upstream URL. It now uses a versioned
pgsty/miniorelease archive verified by its SHA-256 digest, with Go pinned to1.26.5. - Release pipeline verification: a packaging validation workflow compares the binary inside all three package formats byte-for-byte against the build output, and checks package names, checksums, architecture fields, and every metadata field. The expected RPM metadata is sourced from the signing script itself, so configuration drift cannot strand a release part-way through signing.
- CI supply-chain hardening: every GitHub Action is pinned to a commit SHA with dependabot keeping them current, workflow permissions are narrowed to read-only, and a stale workflow pointing at the upstream organization’s project board was removed.
- Documentation: the English and Chinese READMEs now state this fork’s distribution channels and self-update policy explicitly, and installation instructions that would silently install upstream
mcwere removed.
Known issue
mcli watch (bucket event notification) receives no events against any published pgsty/minio server release. The cause is a silent streaming-flush regression on the server side, inherited from upstream — it is not a client problem, and the previous mcli release is affected identically. The fix was merged to the server’s master on 2026-07-29 but has not shipped in a published server release. See the SILO 20260618 release notes and PR #34.
Separately, this fork inherits upstream’s unfixed defects, and with the upstream repository archived they can only ever be fixed here. The most serious is minio/mc#5139: mirror --remove --watch can delete a live object from the target when a non-current version of it is removed from the source. Exercise caution combining --remove --watch on versioned buckets.
Related Commits
- 9603ee3: fix: redact SUBNET secrets in HTTP debug logs
- f6ae2b0: fix: disable self-update in Pigsty builds
- c05a6e4: build: update Go deps and toolchain to 1.26.5
- 1f105aa: build: use local fork artifacts for containers
- 1182da5: ci: pin fork integration test dependencies
- 9ee207f: docs: clarify Pigsty fork distribution channels
- 0686cd8: fix: isolate SUBNET debug redaction
- ad10a2a: build: complete local Docker context isolation
- 5f54221: docs: update mc README and cn version
- 02c0305: build: migrate release packaging to nFPM
- 4c4dcc4: build: harden release provenance and package metadata
2.20 - Silo 20260618 Released
Published: 2026-06-18 · Version: RELEASE.2026-06-18T00-00-00Z
This release is a security and dependency-maintenance update for the pgsty/minio fork. It hardens LDAP STS throttling, completes S3 Select oversized-record enforcement, removes the obsolete ReadMultiple internode storage-REST API, upgrades the Go build baseline to 1.26.4, and refreshes Go module dependencies to pick up additional third-party security fixes.
Note
Known issue: this release — like every earlier community release since RELEASE.2025-12-03T12-00-00Z — carries a silent streaming-flush regression inherited from upstream that breaks mc watch / bucket-notification listeners and S3 Select keep-alives. There is no workaround. The fix was merged to master on 2026-07-29 but has not shipped in a published server release; see PR #34 for the implementation.
Major Changes
- Remove the obsolete
ReadMultiplestorage-REST API: the legacy/rmplinternode endpoint is removed rather than patched in place, including its route, handler, client wrapper, storage interfaces, xlStorage methods, generated datatypes, and related metric. No production caller is expected after upstream multipart handling moved toReadParts, but clusters should still run a consistent release during rolling upgrades. - Complete S3 Select oversized-record enforcement: JSON Lines input now uses the bounded reader path, so oversized records are rejected consistently instead of bypassing limits on SIMD-capable CPUs. S3 Select stream errors now preserve the intended error code and wrap JSON parser failures as
JSONParsingError. - Harden LDAP STS rate-limit source bucketing: throttling is now keyed only by source IP, avoiding username-shared buckets that could be drained by one client to lock out a legitimate user. Trusted-proxy handling now resolves
X-Forwarded-Forfrom right to left, rejects catch-all trusted-proxy CIDRs, ignores RFC 7239Forwarded, and documents theX-Real-IPdeployment contract. - Refresh the Go runtime and module baseline: release, hotfix, goreleaser, and old-CPU Docker builds now use
golang:1.26.4-alpine;go.modis updated to Go1.26.4; and dependencies are refreshed across NATS, Prometheus, Azure SDK, Apache Thrift, gRPC, OpenTelemetry, Google API/auth, Gox/*, and related transitive libraries.
Direct Security Fixes
- CVE-2026-42600: remove the obsolete
ReadMultiplestorage-REST API to close the legacy internode file-read path exposed through/rmpl. - CVE-2026-39414: complete oversized S3 Select record enforcement for JSON Lines inputs and preserve correct S3 Select error semantics.
- CVE-2026-33419: further harden LDAP STS rate-limit accounting and trusted-proxy source-IP handling.
Dependency Security Updates
- Update
github.com/Azure/go-ntlmsspfromv0.1.0tov0.1.1, fixing CVE-2026-32952, where malformed NTLM challenges could panic a Go process. - Update
github.com/apache/thriftfromv0.22.0tov0.23.0, fixing CVE-2026-41602 in the GoTFramedTransportimplementation. - Update
github.com/nats-io/nats-server/v2fromv2.11.1tov2.11.15, absorbing the NATS 2.11.x security patch line. Notable fixes include pre-auth WebSocket and leafnode denial-of-service issues, MQTT authorization issues, JetStream management API authorization hardening, credential exposure fixes, and request identity-spoofing fixes, including CVE-2026-27889, CVE-2026-29785, CVE-2026-33217, CVE-2026-33218, CVE-2026-33222, and CVE-2026-33247. - Update
github.com/prometheus/prometheusfromv0.310.0tov0.311.3, absorbing Prometheus security fixes for remote-read denial of service, stored XSS in UI surfaces, and remote-write configuration secret exposure, including CVE-2026-42154, CVE-2026-44903, CVE-2026-42151, and CVE-2026-40179. - Upgrade the release build baseline through Go
1.26.4and refresh supporting Go module families, includinggolang.org/x/crypto,golang.org/x/net,golang.org/x/sys,golang.org/x/text,google.golang.org/grpc, and OpenTelemetry. These updates keep the fork aligned with patched upstream dependency baselines even where the previously pinned version was already past the specific public advisory range.
Related Commits
2.21 - Silo 20260417 Released
Published: 2026-04-17 · Version: RELEASE.2026-04-17T00-00-00Z
This release focuses on security hardening and compatibility tightening. It bundles fixes across OIDC, LDAP STS, S3 Select, replication metadata handling, unsigned-trailer flows, the Snowball upload path, and multiple dependency- and Go toolchain-related security issues, while also incorporating the LDAP TLS regression fix and a cleanup of community-fork documentation.
Major Changes
- Tighten the identity-authentication flow: OIDC / WebIdentity now accepts only asymmetrically signed
ID Tokenvalues backed by the IdPJWKS; symmetrically signed tokens such asHS256are no longer accepted. LDAP STS also now hides the distinction between unknown-user and bad-password failures to reduce username-enumeration risk. - Update LDAP STS rate limiting: limits now apply to both source IP and normalized username, and successful requests no longer consume quota incorrectly. By default MinIO now uses only the socket peer address as the source and no longer trusts
X-Forwarded-For,X-Real-IP, orForwarded; to rate-limit by real client IP, configureMINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIESexplicitly. - Make upload and write paths stricter: presigned query parameters can no longer be combined with
unsigned-trailerPUTor multipart uploads. Snowball auto-extract now also performs full signature validation on theunsigned-trailerpath and rejects anonymous or forged-signature requests. - Prevent replication metadata spoofing: internal
X-Minio-Replication-*headers attached to ordinaryPUT/COPYrequests are now rejected or ignored, and only trusted replication flows may write the related internal metadata. - Clarify S3 Select error semantics: oversized CSV and line-delimited JSON records now return
OverMaxRecordSizedirectly instead of the genericInternalError; clients or alerting rules that depend on the old error code should be adjusted. - Upgrade the runtime and dependency baseline: fix the regression where
ldaps://did not correctly apply TLS settings, replaceminio/pkg/v3withpgsty/minio-pkg/v3, and pin several critical dependencies that are prone to breaking changes. The release also upgradesgo-jose,go.opentelemetry.io, and Go1.26.2to unify the build and release baseline. - Refresh documentation and security guidance: update
SECURITY.md,VULNERABILITY_REPORT.md,docs/sts/ldap.md, and related documents, add a security advisory index, and switch upstreamminio/minioreferences in the security guidance over topgsty/minio.
Fixed CVEs
- CVE-2026-34986: upgrade
go-josetov4.1.4and fix known security issues in the JWT / JOSE dependency chain. - CVE-2026-39883: upgrade the
go.opentelemetry.iodependency stack to fix the PATH-hijacking risk. - CVE-2026-33322: restore the strict JWKS-only OIDC JWT verification path to block keyring injection and algorithm-confusion risk.
- CVE-2026-33419: systematically harden LDAP STS authentication, rate limiting, source-address identification, and accounting logic across four follow-up fixes.
- CVE-2026-34204: reject injection of
X-Minio-Replication-*metadata by untrusted requests to prevent objects from being written with invalid replication state. - CVE-2026-39414: reject oversized S3 Select records early to avoid continued buffering and parsing of abnormal inputs.
- GHSA-hv4r-mvr4-25vw: close the unsigned-trailer query-auth bypass.
- GHSA-9c4q-hq6p-c237: harden unsigned-trailer authentication and signature validation in Snowball auto-extract scenarios.
- CVE-2026-32280, CVE-2026-32281, and CVE-2026-32283: upgrade Go to
1.26.2and absorb the upstream toolchain and stdlib security fixes.
Related Commits
- ce1c537: fix: pin deps with breaking changes and fix LDAP TLS regression (#15)
- 68e0ba9: fix: upgrade go-jose to v4.1.4 to patch CVE-2026-34986
- 1869bd3: fix: CVE-2026-39883 upgrade go.opentelemetry.io
- ff58df9: merge: PR #18 upgrade go-jose to v4.1.4 for CVE-2026-34986
- e4fa063: merge: PR #19 upgrade go.opentelemetry.io for CVE-2026-39883
- d24f449: fix: CVE-2026-33322 restore JWKS-only OIDC JWT verification
- 3b950f8: fix: CVE-2026-33419 harden LDAP STS auth
- 56fa63b: fix: CVE-2026-34204 reject untrusted replication metadata
- 3252d5b: fix: CVE-2026-39414 reject oversized S3 Select records
- f444b6f37: fix: fake CVE-2026-40027 block unsigned-trailer query auth bypass
- efb6e5b00: fix: fake CVE-2026-40028 harden snowball unsigned-trailer auth
- db4c0fd: fix: CVE-2026-32280/CVE-2026-32281/CVE-2026-32283 upgrade Go to 1.26.2
- 18b712d: fix: CVE-2026-33419 preserve LDAP STS rate limits on success
- 9e10f6d: fix: CVE-2026-33419 harden LDAP STS rate-limit source IP
- f441108: fix: CVE-2026-33419 tighten LDAP STS rate-limit accounting
- f48dbe7: docs: refresh security guidance and fork references
2.22 - Silo 20260325 Released
Published: 2026-03-25 · Version: RELEASE.2026-03-25T00-00-00Z
This is a maintenance release centered on packaging, stability, and security disclosure. It improves the shipping artifacts, fixes an LDAP TLS regression, and explicitly documents the secure dependency set carried by the release.
Major Changes
- Bundle
mcli/mcinto the Docker image and add checksum verification for a better out-of-the-box image experience. - Fix the LDAP TLS regression affecting
ldaps://deployments so TLS settings are correctly honored. - Remove inherited upstream CI/CD workflows that are no longer used in the community-maintained fork.
- Pin several critical dependencies to avoid further fallout from upstream breaking changes.
Fixed CVEs
- CVE-2026-24051: the release notes explicitly call out
go.opentelemetry.io/otel/sdk v1.42.0, which avoids the macOS PATH-hijacking arbitrary code execution issue. - CVE-2025-10543: the release notes explicitly ship
github.com/eclipse/paho.mqtt.golang v1.5.1, fixing incorrect MQTT packet encoding for oversized UTF-8 strings. - CVE-2025-58181: the release notes explicitly ship
golang.org/x/crypto v0.49.0, fixing unbounded memory consumption insshGSSAPI authentication handling.
Related Commits
2.23 - Silo 20260321 Released
Published: 2026-03-21 · Version: RELEASE.2026-03-21T00-00-00Z
This maintenance release is built around the Go 1.26.1 upgrade and a broad dependency refresh. Beyond stricter compiler and linter compatibility fixes, it also delivers the most substantial security dependency refresh in the current release line.
Major Changes
- Upgrade the build environment from Go
1.26.0to Go1.26.1. - Refresh direct and indirect dependencies to converge on the newer toolchain.
- Fix linter and test issues exposed by the stricter Go 1.26.1 checks.
Fixed CVEs
- CVE-2026-27137: Go stdlib
1.26.0->1.26.1fixes incomplete email-constraint enforcement incrypto/x509. - CVE-2026-27138: Go stdlib
1.26.0->1.26.1fixes acrypto/x509panic triggered by malformed certificates. - CVE-2026-25679: Go stdlib
1.26.0->1.26.1fixes insufficient validation of IPv6 host literals innet/url. - CVE-2026-27139: Go stdlib
1.26.0->1.26.1fixesFileInfometadata escaping theRootboundary inos. - CVE-2026-27142: Go stdlib
1.26.0->1.26.1fixes missing URL escaping inhtml/templateformeta refreshcontent. - CVE-2026-26958:
filippo.io/edwards25519v1.1.0->v1.2.0fixes incorrect or undefinedMultiScalarMultbehavior. - CVE-2025-10543:
github.com/eclipse/paho.mqtt.golangv1.5.0->v1.5.1fixes incorrect MQTT packet encoding for oversized UTF-8 strings. - CVE-2026-24051:
go.opentelemetry.io/otel/sdkv1.38.0->v1.42.0fixes the macOS PATH-hijacking arbitrary code execution issue. - CVE-2026-33186:
google.golang.org/grpcv1.77.0->v1.79.3fixes authorization bypass caused by a missing leading slash in the HTTP/2:pathpseudo-header.
Related Commits
2.24 - Silo 20260314 Released
Published: 2026-03-14 · Version: RELEASE.2026-03-14T12-00-00Z
This release switches the project to the community-maintained Console fork and performs a sizeable dependency refresh to establish the base for the later Go 1.26.x maintenance releases.
Major Changes
- Switch to the community-maintained
georgmangold/console v1.9.1fork in place of the unmaintainable upstream Console dependency. - Refresh a large portion of the direct and indirect dependency graph so the new Console and toolchain combination builds cleanly.
- Fix the
go vetformat directive issue ingrid_test.goand adjust tests for the HTTP behavior changes in Go 1.26.
Fixed CVEs
- CVE-2025-47913:
golang.org/x/cryptov0.37.0->v0.46.0fixes a panic inssh/agentwhen handling malformed responses. - CVE-2025-58181:
golang.org/x/cryptov0.37.0->v0.46.0fixes unbounded memory consumption insshGSSAPI authentication parsing. - CVE-2025-47914:
golang.org/x/cryptov0.37.0->v0.46.0fixes a panic inssh/agentcaused by malformed identity messages. - CVE-2025-47911:
golang.org/x/netv0.39.0->v0.48.0fixes quadratic parsing complexity inhtml.Parsefor crafted inputs. - CVE-2025-58190:
golang.org/x/netv0.39.0->v0.48.0fixes an infinite parsing loop ingolang.org/x/net/html.
Related Commits
2.25 - Silo 20260214 Released
Published: 2026-02-14 · Version: RELEASE.2026-02-14T12-00-00Z
This early infrastructure-focused community release restores the embedded Console, introduces GitHub CI/CD, and lifts the Go baseline to 1.26.0, which also absorbs a batch of security fixes from the older toolchain generation.
Major Changes
- Restore the embedded Console and refresh the README to clarify the community fork position.
- Add GitHub CI/CD workflows as the base for automated builds and multi-platform delivery.
- Add quick links for docs, Docker, the GitHub repository, and installation through the
pigpackage manager.
Fixed CVEs
These issues were absorbed as part of the Go 1.25.5 -> 1.26.0 upgrade:
- CVE-2025-68121:
crypto/tlscould incorrectly accept mutated CA configuration during session resumption. - CVE-2025-61730: TLS 1.3 could process handshake messages incorrectly across encryption-level boundaries.
- CVE-2025-61726:
net/urlquery parsing could be abused for memory exhaustion. - CVE-2025-61728:
archive/zipcould consume excessive CPU while building archive indexes. - CVE-2025-68119:
cmd/gocould trigger unexpected code execution when invoking external VCS tooling. - CVE-2025-61731: the
#cgo pkg-config:directive could be abused for arbitrary file writes. - CVE-2025-61732:
cmd/cgocomment parsing discrepancies could enable code smuggling.
Related Commits
- 8630937: Restore embedded console and update README for community fork
- 68521b3: add github ci/cd pipeline
Historical note: CI/CD pipeline evidence now links maintained-history commit 68521b37f. It preserves the pipeline files and includes a README adjustment; this does not claim the full historical commit trees are identical.
2.26 - Silo 20251203 Released
Published: 2025-12-15 · Version: RELEASE.2025-12-03T12-00-00Z
This is the earliest traceable community release. Its purpose is to establish the community packaging and distribution baseline rather than to deliver incremental fixes over an earlier community release.
Major Changes
- Build the community packaging flow around
minio/pkger. - Choose a maintenance-mode upstream MinIO baseline as the starting point for the community-maintained fork.
- Produce the first
apk,deb, andrpmartifacts for ongoing community releases.
Fixed CVEs
- This is the first community release. The GitHub Release does not provide a delta-style security-fix list against an earlier community version, and this page does not attempt to reconstruct the full historical CVE delta against the upstream maintenance baseline.
Related Commits
- d4cd4b4: RELEASE.2025-12-03T12-00-00Z with go 1.25.5
3 - SILO Security Chronicle
This is the security chronicle of the SILO community fork, listed from newest to oldest. Records may cover a CVE, a SILO-local SN finding, several related repairs, or a non-vulnerability correctness audit: the original threat model, the turns taken during review, the rejected alternatives, the final invariant, the evidence, and the compatibility cost all stay with that incident.
The security ledger indexes identifiers, fixed source and first containing releases. An investigation date is not a release date, and publishing an article does not deliver a fix.
3.1 - September Security Repairs: Payload Integrity, IAM Revocation and Console Sharing
This record covers three repairs and their delivery boundaries as of September 16. The identifiers are SILO-local advisory numbers, not CVEs or assigned CVSS scores. The security ledger is the maintained index.
| Finding | Fixed source | Published delivery on September 16 |
|---|---|---|
| SN-2026-012: presigned payload integrity | Server c4b5e1cb4, PR #177 |
Not in Server RELEASE.2026-09-03T13-18-01Z; awaiting a later Server release |
| SN-2026-013: durable IAM revocation | Server PR #191 / #192 | Not in Server 20260903; coordinated upgrade required |
| SN-2026-014: anonymous Console share proxy | Console PR #56; Server PR #209 selects the repair | Standalone Console v2.4.1 is published; the embedded copy still requires a new Server binary |
Presigned payload integrity — SN-2026-012
The generic authenticated path could verify a presigned request using a payload
hash supplied only in X-Amz-Content-Sha256, yet omit checking the actual body
against that hash. A holder of a suitably signed write URL could replace its
body. The committed regression demonstrates this with PutBucketPolicy;
this is not a claim that an anonymous caller can write arbitrary bucket policies.
c4b5e1cb4 in
PR #177 binds body validation to the
effective signed hash; mismatches fail with XAmzContentSHA256Mismatch (400).
An explicit signed UNSIGNED-PAYLOAD retains its intended meaning. The adjacent
signed-field fixes align policy conditions with authenticated values and remove
the X-Amz-Signature-Age scratch header. They complement
SN-2026-011; they do not establish complete
unsigned-header coverage for streaming SigV4.
Review custom signing clients before upgrading. Query hash values take precedence over the fallback header, so changing an unsigned header must not change what policy evaluation or body validation sees.
Durable IAM revocation — SN-2026-013
Delayed replication events, parent recreation and lost deletion history could restore identities or grants an operator had revoked. PR #191 and PR #192 retain source revisions, deletion tombstones and signed parent-revocation boundaries. Older events and child credentials cannot silently cross a retained revocation boundary.
This is a persistent-state change. Upgrade all sites and all nodes sharing an
IAM backend together; mixed old/new nodes and rolling downgrade are unsupported.
mcli admin cluster iam export omits deletion history and is not a complete
recovery backup. Preserve a tested full-backend recovery point and reconcile
known earlier revocations. Tombstones have no TTL or automatic compaction.
The design record documents ordering, remaining group-membership limitations and metrics. Follow the upgrade and recovery procedure, including the password-policy pre-step. A revocation can commit and then return HTTP 500 if cleanup fails; the response alone does not prove that the old credential is valid.
Anonymous Console share proxy — SN-2026-014
Jiri Pejchal reported that the unauthenticated shared-download proxy accepted URLs beyond the intended object-download surface on its configured Server origin. This is a same-origin proxy-boundary issue; it is not evidence of arbitrary-host SSRF or bypass of the Server’s S3 authorization.
Console #56 constrains scheme, host and port, requires a valid bucket/object path, rejects system and traversal paths and operation-changing query selectors, disables redirects, and propagates caller cancellation. Accepted object URLs retain their original signed bytes.
Upgrade a standalone installation to Console v2.4.1. Server #209 selects the fixed Console source on main, but installing a separate Console cannot patch the UI and proxy compiled into Server 20260903. Until an appropriate Server artifact is available, limit exposure of the affected Console share endpoint according to the deployment.
Adjacent dependency hardening
Server main upgrades amqp091-go to v1.14.0 for
GHSA-6c5v-hqjr-5xxp, fixed
upstream in v1.13.0. A malicious AMQP peer can trigger excessive allocation;
the SILO path requires a configured AMQP notification target. This Server change
also awaits a release after 20260903. pkg v3.14.1
separately ships the JWX JSON encoding update. Component publication and Server
dependency adoption are separate facts.
3.2 - SN-2026-011: Fix and Release Status
Status on 2026-09-13: SN-2026-011 is fixed on Server main, starting with
123325430.
The latest published Server, RELEASE.2026-09-03T13-18-01Z, and earlier public
Server releases are affected. No new fixed Server release is established by
the pkg v3.14.0 or mcli 20260913 publication.
A holder of a signed PUT request could add an unsigned x-amz-copy-source
header and turn the permitted write into a copy of another object readable by
the signing key. A destination that permits anonymous reads can expose those
copied bytes. This affects both presigned and Authorization-header requests;
the holder does not need the signing credentials.
The patch checks the received x-amz-* headers against the signed-header set,
with the protocol’s explicit exceptions, before dispatching the requested
operation. Follow-up request-signing and checksum regressions are documented in
the signed-header review.
Operators must update the Server to source containing the fix or a future release that explicitly includes it. Restrict write-signing credentials to the required objects and avoid exposing unnecessary read grants or anonymous readable upload destinations while planning that update. Updating a client or Console alone does not remove the Server defect.
Reported by Oren Yomtov. The advisory ledger records SN-2026-011 and its source fix; a CVE was requested. Do not substitute a dependency scan’s clean reachability result for this application-level status. See the component matrix for released versus main source and the Server changelog for the remaining release contents.
3.3 - CVE-2025-62506: Session-Policy Privilege Escalation
Status: Inherited and released
First Silo community release: RELEASE.2025-12-03T12-00-00Z
Upstream fixed release: RELEASE.2025-10-15T17-29-55Z
GitHub advisory: GHSA-jjjj-jwhf-8rgr
Upstream fix: minio/minio#21642
A service account or STS account with a restricted session policy could use an “own account” operation to create another service account without the restriction. The child account then inherited broader parent permissions, turning valid low-privilege credentials into a privilege-escalation path.
Silo did not need a separate backport. The upstream fix and its two regression groups were already in the history from which the community fork was released. This note records that inheritance without duplicating the tests or reconstructing the investigation already captured by the upstream advisory and PR.
Commit mapping
| Role | Repository | Commit | Evidence |
|---|---|---|---|
| Upstream remediation | minio/minio |
c1a49490 |
Merge commit for PR #21642 |
| Silo inheritance | pgsty/silo |
c1a49490 |
The fork preserves the same commit object and SHA; the first Silo community release descends from it |
The identical SHA is the mapping: this is inherited source history, not an independent Silo implementation with a merely similar patch. The advisory ledger keeps the same source-to-fork record.
Regression evidence
PR #21642 added two regression groups and runs each against root and non-root parents:
TestServiceAccountPrivilegeEscalationBug2_2025_10_15covers restricted service accounts.TestSTSPrivilegeEscalationBug2_2025_10_15covers restricted STS accounts.
Both remain wired into the standard cmd test suites, so go test ./cmd executes them. No parallel or replacement regression test was added for this publication update.
Fix and operator boundary
When a session policy exists, the fix clears the DenyOnly shortcut before evaluating it. An “own account” exception therefore cannot turn “not explicitly denied” into permission: the restricted policy must actually allow the action.
Every Silo community release contains this commit. Operators migrating from a MinIO build older than the upstream fixed release should upgrade, review service accounts created by restricted service or STS identities, and revoke suspicious child accounts. This inheritance statement does not extend support to historical Silo releases; the current release line remains the supported line.
3.4 - SILO 20260903 Security Notes: SN-2026-006 through 010
Released in SILO 20260903. These fixes are part of
RELEASE.2026-09-03T13-18-01Z. See the complete release notes for the 20260806-to-20260903 upgrade boundary, components, verification evidence, and known deferrals.
This bulletin collects five related security findings from the final SILO Server review. They cover customer-provided encryption keys, client-controlled replication headers, administrative status changes, and explicit object-version deletion.
The identifiers SN-2026-006 through SN-2026-010 are fork-local security-note IDs. They are not CVEs, are not registered in a vulnerability database, and should not be represented as such. The individual CVE chronicle remains one article per CVE; this release bulletin groups findings without CVEs so operators can evaluate one upgrade boundary.
All five defects were inherited from the archived minio/minio lineage and affect every earlier SILO release. “Inherited” describes provenance, not severity.
Summary
| ID | Required attacker position | Security failure | Released behavior |
|---|---|---|---|
SN-2026-006 |
Read access to a zero-byte SSE-C object | A wrong customer key was accepted because no data block was decrypted | The object key is authenticated; wrong keys return 403 AccessDenied |
SN-2026-007 |
Read access to an SSE-C object, or a client-supplied replication marker | GetObjectAttributes returned protected metadata without authenticating the customer key |
Ordinary requests need the correct key; only an authorized replication peer uses the replica exception |
SN-2026-008 |
Any authenticated principal able to invoke the affected read/write/delete operation | Internal-looking headers granted replication-only effects without replication authorization | Exact marker plus s3:ReplicateObject or s3:ReplicateDelete is required |
SN-2026-009 |
Authenticated admin API principal holding only one status action | The enable action authorized disable, and vice versa | The requested target state selects the required admin action |
SN-2026-010 |
Authenticated S3 principal with s3:DeleteObject but not s3:DeleteObjectVersion |
A caller could permanently delete an explicit version without the AWS-required action | Explicit version deletion requires s3:DeleteObjectVersion |
No finding in this bulletin is an unauthenticated remote-code-execution claim. Each requires an authenticated identity or existing object permission, but each crosses a privilege or cryptographic boundary beyond that identity’s intended grant.
SN-2026-006: zero-byte SSE-C key authentication
Affected operations: GetObject, HeadObject, CopyObject source processing, and GetObjectAttributes for zero-byte objects encrypted with SSE-C.
Tracking: issue #82.
Fixes: b73581b05 and c4fd97d0b.
SSE-C stores a sealed object key and requires the caller to provide the customer key again on reads. Normal reads authenticate that key while preparing the encrypted stream. A zero-byte object has no payload block, and the old fast path returned before unsealing the stored object key.
Consequently, an incorrect customer key could receive a successful response. A copy or new version could also be created under a key chosen by the caller without proving knowledge of the current key. The caller still needed read access to the object; the failure was that object permission replaced the separate cryptographic proof.
The fix authenticates the key independent of payload length. Correct keys behave as before. Wrong keys fail with 403 AccessDenied, including null-version and key-rotation cases.
SN-2026-007: SSE-C GetObjectAttributes
Affected operation: GetObjectAttributes on SSE-C objects.
Tracking: issue #84.
Fixes: 474cd5801, 74c97d005, and 21870fa2e.
GetObjectAttributes exposes object size, ETag, checksums, storage class, and multipart part metadata. The inherited handler returned those attributes without authenticating the supplied SSE-C key. It also treated the presence of X-Minio-Source-Replication-Request as permission to skip key handling.
The repair separates two cases:
- an ordinary S3 caller must supply the correct SSE-C key;
- a real replication peer may use the replica path only after authentication and
s3:ReplicateObjectauthorization.
A wrong key now returns 403. A bare marker without the required key and permission cannot create the replica exception. Existing replication identities that already hold the required action keep working.
SN-2026-008: replication headers are not authority
Affected surface: object reads and writes, multipart completion, object and version deletion, Snowball extraction, Object Lock timestamps, checksum metadata, and bucket events.
Tracking: PR #101.
Primary fix range: 938603458 through 04b097fd9, followed by Snowball and rule-prefix hardening.
CVE-2026-34204 stopped ordinary PUT and COPY requests from importing a subset of replication SSE metadata. The wider audit found that many other consumers still treated header presence as proof that a request was internal replication.
Depending on the operation, a client holding ordinary read or write permission could:
- request an SSE-C ciphertext/no-decryption path without the customer key;
- preserve a supplied source ETag or modification time;
- inject source checksum, actual-size, retention, legal-hold, or replica-state metadata;
- choose replication-only delete semantics;
- suppress successful object events;
- carry trust from one Snowball archive entry into another.
The repair defines one receiver-wide authority:
- authenticate the original request in its signed form;
- require exactly one marker with the exact lowercase value
true; - authorize
s3:ReplicateObjectors3:ReplicateDeleteon the addressed resource; - derive a narrower replica-trusted state only where replica status also proves it;
- store the decision in private request context;
- strip untrusted internal headers from a clone after authentication.
The context decision is authoritative. Stripping protects older consumers but cannot itself grant trust. The clone preserves the original trailer map so streaming checksum authentication remains correct.
The detailed protocol matrix, CORS interaction, black-box replication test, and rejected alternatives are in No I/O Before Auth, No Privilege From Headers.
SN-2026-009: user and group status authorization
Affected operations: admin SetUserStatus and SetGroupStatus.
Tracking: PR #73 / PR #85.
Fixes: 58735ee38 and 229fe2b3c.
The inherited handlers checked admin:EnableUser or admin:EnableGroup regardless of the target state. A narrowly delegated administrator permitted to enable an identity could therefore disable it too; an identity intended to hold only the disable action could fail or be evaluated against the wrong grant.
The requested state now determines the required action:
| Target | Required action |
|---|---|
| Enable user | admin:EnableUser |
| Disable user | admin:DisableUser |
| Enable group | admin:EnableGroup |
| Disable group | admin:DisableGroup |
admin:* and the built-in consoleAdmin policy remain sufficient. Only custom least-privilege policies that accidentally relied on one action for both directions need adjustment.
SN-2026-010: explicit object-version deletion
Affected operations: DeleteObject and each DeleteObjects entry carrying an explicit versionId.
Tracking: issue #58 and PR #104.
Fix range: 75a6734e4 through d2d47a41f.
The inherited S3 path authorized explicit version deletion as ordinary s3:DeleteObject and used s3:DeleteObjectVersion only as a secondary deny check. This diverged from AWS and let a principal with ordinary delete permission permanently remove a selected historical version.
The release requires s3:DeleteObjectVersion for explicit versions. A delete without versionId continues to use s3:DeleteObject. Replication targets keep their s3:ReplicateDelete contract.
There are two policy effects:
- a principal granted only
s3:DeleteObjectcan no longer delete explicit versions; - a policy using
Allow s3:*together withDeny s3:DeleteObjectto block permanent deletion must also denys3:DeleteObjectVersion.
The multi-delete repair preserves authentication and audit context for every entry and has least-privilege tests for ordinary and replication deletes.
Operator actions
Before upgrading:
- identify policies and applications that delete explicit versions; grant or deny
s3:DeleteObjectVersionintentionally; - inspect custom admin policies for enable/disable user and group operations;
- confirm that replication service accounts have only the required
s3:ReplicateObject/s3:ReplicateDeleteactions and that ordinary application identities do not; - verify SSE-C integrations send the same customer key for GET, HEAD, attributes, and copy-source operations, including zero-byte objects;
- if an application was relying on a wrong SSE-C key returning success, treat that as a latent client defect and correct it before rollout.
After upgrading:
- exercise a wrong-key negative test and a correct-key positive test on a disposable SSE-C object;
- test one normal replication and one delete replication operation;
- test each delegated admin direction independently;
- test version deletion with a principal that has only
s3:DeleteObjectand with one that also hass3:DeleteObjectVersion; - review audit logs for unexpected denied operations instead of widening policies immediately.
Verification boundary
The fixes have focused unit, handler, authorization, streaming-trailer, Snowball, and site-replication tests. Full local acceptance on ebac0ca73 included the complete cmd race suite and six deployment shapes with 174 PASS / 0 FAIL. The final line adds one CORS state-clear fix after that run and passes targeted race, compatibility, generation, diff, and lint gates.
The final RELEASE.2026-09-03T13-18-01Z publication completed the gates recorded in the pre-release review: exact-SHA CI and Test Release, signed packages, checksums, SBOMs, provenance, and public classic/distroless container images. This bulletin still scopes its claims to the five fixes above and the residual risks below.
Residual risks outside this bulletin
- Conditional deletion is still unsupported:
DeleteObjectignoresIf-MatchandDeleteObjectsignores per-object ETag (#10). - Non-CORS policy/SSE/tag/quota deletions may not converge across site-replication peers (#77).
- The trust audit focused on known replication and SSE-C fields. New internal headers must still answer the same provenance-and-authorization question during review.
The release notes contain the full known-issue and deployment boundary.
3.5 - CVE-2026-32285: The jsonparser Advisory That Required No Patch
Status: Closed without a code change
GitHub issue: pgsty/minio#26
Security maintenance is not always a sequence of “find a vulnerability, then ship a patch.” The initial assessment of CVE-2026-32285 was that the repository might still carry a vulnerable jsonparser; replacing the dependency or maintaining another fork was even considered. Checking the resolved module version and actual reachability changed the conclusion: the tree already used v1.1.2, which contained the fix, and govulncheck found no reachable vulnerable symbol.
The right final action was not to manufacture an upgrade. It was to record the evidence and close the issue.
What was wrong with the initial assumption
The issue was first understood as “this dependency has no fixed version.” Acting on that premise without verification could have produced several changes that looked proactive but made the project worse:
- changing the dependency graph for no security benefit;
- introducing compatibility regressions in the name of a nonexistent fix;
- adding another fork that would need long-term maintenance;
- implying that previous SILO releases were demonstrably exposed when that had not been established.
Security work cannot be measured by whether it produces a diff. Leaving correct code unchanged is itself a security decision, and it needs evidence.
Investigation
The investigation narrowed the question through four layers of evidence:
- Confirm the version actually selected in the current
go.modandgo.sumgraph. - Check the upstream release and establish that
v1.1.2already contained the relevant fix. - Run and inspect
govulncheck; it reported no reachable vulnerable symbol. - Attribute the discrepancy in the issue to stale advisory or vulnerability-database information, not to a vulnerability still present in the source tree.
Four claims must remain separate: a version was once listed as affected, a package is imported, a vulnerable symbol is reachable in the program, and remote input can actually exploit that path. None of them proves the others.
Why there was no “just in case” upgrade
If the selected version already includes the fix, bumping to an arbitrary newer version does not make the system safer. It only expands the change surface and makes later regressions harder to attribute. That is especially risky in a large Go module graph.
The final decision was therefore to:
- avoid committing a fictitious fix;
- preserve the version and reachability evidence in the issue;
- keep version gates and
govulncheckin place to detect a future dependency rollback; - treat “no change required now” as a dated conclusion, not a permanent exemption.
Verification boundary
This incident established that the checkout examined on 2026-04-15 did not require a code change for CVE-2026-32285. It does not establish that every future branch, module graph, or release will remain unaffected. A dependency downgrade or a change in module selection requires the version and reachability checks to be repeated.
This article records the investigation and the basis for closure. The original govulncheck was not rerun while preparing this chronicle.
The principle this incident left behind
The objective of security maintenance is an accurate risk conclusion, not a patch for every issue. For a dependency CVE, ask in order: which version is actually selected, whether the vulnerable code enters the program, whether the symbol is reachable, and whether a deployed entry point makes it exploitable. Only when those answers require a source change should the investigation produce a diff.
3.6 - CVE-2026-33322: OIDC JWT Algorithm Confusion
Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Affected entry points: AssumeRoleWithWebIdentity, AssumeRoleWithClientGrants
GitHub issue: pgsty/minio#22
The old implementation placed the OIDC client secret in the JWT verifier keyring while also accepting HMAC signing methods. An attacker who knew that client secret could therefore mint an HS-signed token and exchange it through STS for temporary credentials. The final fix restored asymmetric, JWKS-only verification. It deliberately broke HS256/384/512 compatibility instead of keeping an option that would reintroduce ambiguous trust semantics.
The vulnerability was not the absence of signature verification
At first glance, the old code did verify JWT signatures. The boundary that failed was more specific: which kind of key the verifier would accept, and whether the token header could select an algorithm with semantics that did not match that key’s intended role.
The attack chain required several conditions:
- the attacker obtained the OIDC client secret;
- the attacker constructed an HMAC-signed ID token;
- the verifier treated the client secret as an HMAC signing key;
- the token reached the WebIdentity or ClientGrants STS flow and was exchanged for temporary credentials.
Disclosure of a client secret is already serious, but it should not automatically confer the power to issue arbitrary user ID tokens. Combining those capabilities in one keyring created the algorithm-confusion vulnerability.
A compatibility path was implemented, then deliberately removed
During the fix, an allow_hmac-style compatibility path was implemented. It appeared reasonable: keep the secure default while letting users with a real requirement opt in. But retaining a shared secret in the general verifier keyring meant administrators would need to understand that the option expanded the entire STS trust boundary. Any future drift in the method allowlist could reopen the flaw.
The trade-off became clear:
| Option | Benefit | Risk | Decision |
|---|---|---|---|
| Keep the secret keyring and restrict some algorithms | Small change; preserves HMAC IdPs | The keyring still mixes two trust semantics | Rejected |
Add an allow_hmac option |
Makes compatibility explicit | The option is difficult to reason about correctly and expands the test surface | Implemented, then reverted |
| JWKS-only verification | Clear boundary; refresh and retry use the same parser | HS users must migrate | Accepted |
The most important decision was not what code was added, but that a completed compatibility implementation was removed.
Final invariants
The fix was concentrated in the OIDC JWT verification path and established four rules:
- verifier keys come only from the identity provider’s JWKS;
- the OIDC client secret never enters the JWT verification keyring;
- HS256, HS384, and HS512 are always rejected;
- the ordinary RS256 flow and JWKS refresh/retry use the same method allowlist.
The CVE was not used as a pretext for expanding JOSE support. PS256 and EdDSA remained out of scope.
Verification and release
The development record includes HS256 rejection, RS256 acceptance, JWKS refresh/retry regression tests, and focused go test ./internal/config/identity/openid. Temporary compatibility helpers, configuration, and tests were all removed from the final diff.
The public fix is d24f449, released with SILO 2026-04-17. This article records the historical verification; those tests were not rerun while preparing the chronicle.
Compatibility cost
This is an explicit breaking change. Identity providers that still issue HS256/384/512 tokens must migrate to JWKS-backed RSA or ECDSA before upgrading SILO. The project chose a narrower trust model that is easier to explain and audit over preserving a configuration that happened to work before.
3.7 - CVE-2026-33419: LDAP STS Enumeration and the Throttling Chain
Status: Released, followed by two rounds of corrections
First containing release: RELEASE.2026-04-17T00-00-00Z
Complete correction: RELEASE.2026-06-18T00-00-00Z
GitHub issue: pgsty/minio#23
The core vulnerability was straightforward: LDAP STS returned different results for “user does not exist” and “password is wrong,” creating a username oracle. The first fix unified the external authentication failure and added limits by source IP and username. Continued review then showed that success refunds, spoofable source headers, reservation accounting, and the shared username bucket could turn the security control itself into a new attack surface.
The final June design removed the username bucket that enabled precise account lockout, retained only the source-IP bucket, and made proxy attribution an explicit deployment contract.
Initial threat model
The entry point is AssumeRoleWithLDAPIdentity. An attacker needs no existing MinIO account. Access to the LDAP STS endpoint is enough to compare the code, status, or message returned for an unknown user and a wrong password, enumerate valid usernames, and combine that knowledge with password spraying, guesses about organizational naming, or social engineering.
The fix could not simply disguise every error as “wrong password.” LDAP connection, lookup-bind, and directory-service failures still needed to appear as infrastructure errors, or operators would lose the ability to diagnose the service.
First round: uniform responses and a limiter
The initial fix on 2026-04-15 did three things:
- unknown user and bad password returned the same external STS authentication error;
- LDAP infrastructure failures still returned 500, with the real cause retained in the server log;
- a new in-memory limiter initially created buckets for both source IP and normalized username.
This closed the content side channel and raised the cost of brute-force attempts, but the limiter state machine and source attribution exposed more problems.
Second round: success, attribution, and accounting
The follow-up changes on April 16 addressed three classes of defects:
- Successful authentication must not consume the failure allowance; the reserve/commit/cancel/refund lifecycle had to be explicit.
- The socket peer must be used by default;
X-Forwarded-For,X-Real-IP, andForwardedcannot be trusted merely because a request supplies them. - Refund and capacity need hard bounds so cancel logic cannot mint tokens.
A proxy must be placed on an explicit trusted allowlist before it can influence source attribution.
Third round: remove the username bucket
Adversarial review in June overturned the intuition that “source plus username must be stronger than source alone.” A shared username bucket could be exhausted continuously from arbitrary origins. With only a low request rate, an attacker could lock a targeted account before the legitimate user ever reached an LDAP bind.
The final fix therefore:
- removed the per-username bucket;
- peeled trusted hops from XFF right to left and selected the first untrusted address;
- rejected trusted-proxy footguns such as
0.0.0.0/0and::/0; - stopped using
Forwardedfor security-sensitive bucketing; - allowed
X-Real-IPonly under a contract in which the proxy overwrites rather than forwards client input.
This turn in the review shows that a security control needs its own threat model. More dimensions of throttling do not automatically mean more security.
Rejected alternatives
| Option | Why it was rejected |
|---|---|
| Perform a dummy bind for unknown users | Amplifies LDAP load and creates a second, error-prone authentication path after the content side channel is already closed |
Bucket all IPv6 clients by /64 |
Legitimate users behind the same site or carrier prefix can throttle one another |
| Take the leftmost XFF value | Client-controlled and therefore spoofable |
Fall back to the peer when XFF and X-Real-IP disagree |
An attacker can create disagreement deliberately and collapse every user behind a proxy into one bucket |
Fully support RFC 7239 Forwarded |
Security-sensitive parsing complexity outweighs the practical benefit |
Verification and release
The historical record covers limiter reserve/commit/cancel/refund behavior, concurrency, success and infrastructure failures, external equivalence of unknown-user and bad-password responses, and RemoteAddr, spoofed-header, trusted-proxy, multi-hop, and catch-all-CIDR cases. Focused package tests and builds were recorded as passing.
The LDAP security end-to-end test skips when _MINIO_LDAP_TEST_SERVER is absent, so an outer ok cannot be presented as proof of the full LDAP scenario.
The first public fix was 3b950f8. Follow-up corrections include 18b712d, 9e10f6d, f441108, and 5e40665.
Final cost and residual risk
- The limiter now keys only on source IP and gives up a hard per-account throttle across different origins.
- It is per-node and in-memory, not a cluster-wide password defense.
- Botnets, distributed origins, IPv6 address rotation, and LDAP bind timing remain.
- Incorrect trusted-proxy configuration can still destroy source attribution.
- A
Forwarded-only deployment falls back to the peer bucket and loses granularity.
Rate limiting can reduce attempts from one source. The uniform external authentication response is what actually conceals whether a username exists.
3.8 - CVE-2026-34204: Replication Metadata Injection
Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
GitHub issue: pgsty/minio#24
Ordinary PUT and COPY requests could smuggle X-Minio-Replication-* headers into internal X-Minio-Internal-* SSE metadata, creating objects whose replication state did not match the authorized path and could even make them unreadable. The final fix stopped accepting replication-only metadata by default, restored it only in a trusted flow authorized for ReplicateObjectAction, and sanitized CopyObject before any header consumer ran.
Threat model
An attacker needed only ordinary object-write permission, not internode credentials. The input came entirely from client-controlled X-Minio-Replication-* headers, but metadata extraction converted it into internal replication or SSE state.
Later read paths interpreted the object according to that false internal state. The result could be an unreadable object: an integrity and availability failure. Almost every production server accepting untrusted writes needed to be treated as affected.
The root problem was not the header name. It was that data from an untrusted source acquired internal semantics without passing replication authorization.
Reject the whole request, or sanitize precisely?
Rejecting an ordinary request whenever it contains a replication header is the most obvious fix. It would also turn a header clients were previously allowed to send and have ignored into a hard failure. The final design was more precise:
- the default extraction path does not accept replication-only metadata;
- ordinary
PUTandCOPYstrip those fields first; - only a path authorized for
ReplicateObjectActionrestores them; - replica-status writes use the same trusted condition;
- legitimate multipart and Snowball replication flows explicitly restore the SSE metadata they require.
That keeps the compatibility change inside internal semantics instead of expanding it to every client carrying an extraneous header.
Why CopyObject had to sanitize early
CopyObject headers are not used only for the final metadata map. They can be consumed earlier by precondition logic and SSE-C source handling. Removing them immediately before the object write is too late: earlier consumers have already been contaminated.
The final sanitization occurs before those consumers. “Untrusted replication headers never enter internal semantics” becomes one invariant instead of a convention every downstream function must remember to enforce.
Implementation and verification
The change covered handler utilities, object handlers, and multipart handlers, with tests at several layers:
- trusted and untrusted metadata extraction at the helper layer;
- malicious
PUTandCOPYcases at the handler layer; CopyObjectheader sanitization;- red/green comparison between the vulnerable parent and the patched tree;
- live-server before/after behavior showing that a malicious header no longer made an object unreadable;
- continued operation of legitimate replication, multipart, and Snowball flows.
The public fix is 56fa63b. This article preserves the historical verification boundary; no live server was started again while preparing the chronicle.
Cost and residual risk
- Internal replication headers supplied by ordinary clients are now ignored or stripped.
- Replication-only metadata must be restored explicitly inside an authorized branch.
- If a future replication entry point forgets to restore it, the result should be a functional regression rather than another untrusted write path.
- The audit focused on replication headers; it does not establish that every
X-Minio-Internal-*field has undergone the same trust review.
This incident leaves a simple review question: a field that looks “internal” is not necessarily trusted. Ask where it came from and which authorization decision allowed it to acquire internal meaning.
The later CORS and replication trust design record generalizes that lesson into one receiver-wide authority for SSE-C reads, source timestamps, multipart completion, object lock, deletes, events, and pre-authentication CORS work.
3.9 - CVE-2026-39414: Oversized S3 Select Records and a SIMD Bypass
Status: Released; the second-round fix was completed in June
Initial fix release: RELEASE.2026-04-17T00-00-00Z
Complete fix release: RELEASE.2026-06-18T00-00-00Z
GitHub issue: pgsty/minio#25
The first fix in April reused the existing 1 MiB maxCharsPerRecord limit for both CSV and ordinary JSON Lines. This prevented unbounded buffering while waiting for a delimiter and returned the explicit OverMaxRecordSize error to clients. A June review then found that CPUs with SIMD support took a different simdjson fast path that bypassed the limit completely.
The final solution sent JSON Lines through the bounded reader on every CPU. The same review also corrected error mapping, parser errors, and the flush of completed records before a terminal error. SILO temporarily gave up the SIMD fast path in exchange for consistent security semantics.
Threat model
An attacker can submit or query an object containing an extremely long single record. The reader continues buffering until it sees a record delimiter, allowing memory and CPU denial of service. More subtly, the same input can select a different implementation according to the machine’s CPU features. Safe behavior on a test machine does not necessarily prove safe behavior in production.
Error semantics are part of the fix. If an oversized record appears only as a generic InternalError, clients and monitoring systems cannot distinguish an enforced security limit from a server failure.
First round: reuse the existing 1 MiB invariant
The first patch did not invent a new configuration knob. It applied the existing maxCharsPerRecord = 1 MiB rule:
- the CSV splitter and line-delimited JSON rejected oversized records before buffering or parsing them further;
- the earliest splitter error was preserved instead of being overwritten by a partial decode;
- the error propagated as
OverMaxRecordSizerather than collapsing intoInternalError.
This was a deliberate compatibility contraction. Clients with lines or records larger than 1 MiB now had to split their input.
Second round: a hardware-dependent bypass
Following the call chain again in June exposed this path:
When simdjson.SupportedCPU() returned true, JSON Lines bypassed the bounded json.PReader. The third-party parser kept reading past a chunk boundary until it found a newline. A generic reader wrapper could not simultaneously preserve already completed records and guarantee a bound on the next record.
The final choice was not another wrapper. JSON Lines temporarily stopped using the SIMD path and always used the bounded PReader. If SIMD support returns, that implementation must enforce the same record limit itself and pass the same CPU-independent regression suite.
Stream semantics corrected in the same round
The review also fixed several adjacent behaviors:
- use
errors.Asto pass through errors implementingSelectError, not just one concrete type; - have the JSON worker wrap parser failures as
JSONParsingError; - flush completed records still waiting below the batch threshold before emitting a terminal error event;
- preserve error priority in input order instead of letting a later oversized record overwrite an earlier parse error.
Those details determine whether a client sees the correct failure or a resource-limit fix that quietly broke the streaming protocol.
Deliberately left outside this CVE
- The historical mismatch between CSV
AllowQuotedRecordDelimiterand the outer physical-newline splitter. - Whether
\rin CRLF counts toward the record length. - Restoring SIMD performance without an equivalent bound.
These questions may be real, but they require independent AWS-compatibility evidence or a more complex quote-aware splitter. They did not belong in a security patch based on guesses.
Verification and release
The historical record includes oversized JSON Lines, error-code preservation, and behavior tests that do not depend on the local machine’s SIMD capabilities. go test ./internal/s3select/... -count=1 and git diff --check were recorded as passing.
The initial public fix was 3252d5b; the complete June correction is fd69c89. Those tests were not rerun while preparing this article.
Final trade-offs
- JSON Lines performance may decrease; this incident did not produce a benchmark that quantifies it.
- The 1 MiB per-record limit rejects oversized input accepted by previous releases.
- Quoted, multiline CSV semantics still need separate work.
- Any future CPU-specific fast path must share the slow path’s security tests.
The second fix leaves the central lesson: a security invariant must hold across hardware-dependent paths. A green test on one CPU does not prove that another execution engine is protected.
3.10 - CVE-2026-40344: Snowball Auto-Extract Authentication Bypass
Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
GitHub advisory: GHSA-9c4q-hq6p-c237
Snowball’s PutObjectExtractHandler omitted the streaming unsigned-trailer authentication case. A tar stream with a forged signature could enter untar() before authentication completed, and one request could fan out into many object writes. The final fix initialized the correct reader, handled the decoded length, and completed SigV4 verification before any tar byte reached the extractor.
Why the identifier changed
The official CVE had not been assigned when the fix was written, so the commit subject used the temporary identifier fake CVE-2026-40028. The final identifier is CVE-2026-40344. The historical commit was not rewritten; the advisory and this article use the official number.
From one missing authentication case to bulk object writes
The entry point was Snowball / PutObjectExtract auto-extraction. The request used unsigned-trailer streaming, an authentication type the old handler did not cover as ordinary PUT did.
The danger was larger than one incorrectly authorized request. Once the tar stream entered untar(), that request could create multiple attacker-chosen objects. An authentication omission was therefore amplified into a bulk-write problem.
Final invariant: the extractor sees zero bytes on failure
The key statement in the fix was:
If authentication ultimately fails,
untar()must have seen zero bytes.
That rule excludes “extract first, then roll back if authentication fails.” Object writes travel through several paths, and proving a complete rollback is much harder than proving that input never crossed the boundary. Authentication had to close before data entered the extractor.
Implementation
The final change:
- recognized
authTypeStreamingUnsignedTrailer; - read
X-Amz-Decoded-Content-Length; - used
newUnsignedV4ChunkedReader(); - performed complete SigV4 request verification before entering
untar(); - preserved valid signed Snowball requests and CRC32 trailer flows.
Verification
The historical commit and investigation record cover:
- rejection of a forged-signature Snowball unsigned-trailer request;
- rejection of anonymous Snowball writes to a non-public bucket;
- successful extraction with a valid signature and trailing CRC32;
- red/green comparison between the vulnerable parent and the patched tree;
- containerized before/after smoke tests.
The public fix is efb6e5b00. The container tests were not rerun while preparing this article.
Compatibility and residual risk
- Clients that relied on an unsigned-trailer Snowball combination whose signature was never really verified will fail after upgrading.
- Authentication now closes before extraction, but tar paths, archive-size limits, and object-count limits remain separate security surfaces.
- Snowball and ordinary unsigned-trailer requests now share a reader; future changes must regress both paths together.
The essence of the fix was not another if. It moved the authentication decision in front of the actual amplification boundary.
3.11 - CVE-2026-41145: Unsigned-Trailer Query Authentication Bypass
Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
GitHub advisory: GHSA-hv4r-mvr4-25vw
Query-string SigV4 credentials could enter a STREAMING-UNSIGNED-PAYLOAD-TRAILER data flow, while the old code verified the signature only when an Authorization header was present. A request carrying a valid access-key identifier could therefore complete a write without a correct signature.
The final fix moved presigned rejection and SigV4 verification into newUnsignedV4ChunkedReader(), making every caller consuming that stream share one authentication boundary.
Identifier note
The official CVE had not been assigned when the patch was written, so its commit subject used fake CVE-2026-40027. The final identifier is CVE-2026-41145. The historical commit remains unchanged; public material uses the official identifier.
Root cause: authentication was coupled to transport form
The affected entry points included PutObject and PutObjectPart. The request selected STREAMING-UNSIGNED-PAYLOAD-TRAILER, with its credentials and signature in the query string rather than the Authorization header.
The old handler used header presence to decide whether to verify a signature. The body reader still consumed the data normally, silently degrading query authentication into something close to an anonymous write. The attacker needed to know a valid access-key identifier, but did not need to produce a correct signature.
The problem was not failure to parse the query parameters. It was that authentication depended on how credentials were transported instead of the trust boundary at which the stream was consumed.
Why the patch did not live in each handler
| Option | Risk | Decision |
|---|---|---|
Add header/query checks separately to PutObject and PutObjectPart |
Closes today’s entry points, but a new caller can omit the check again | Rejected |
| Invent a compatible presigned unsigned-trailer protocol | Greatly expands protocol and test surface without an existing support contract | Rejected |
Reject and verify centrally in newUnsignedV4ChunkedReader() |
Forces every consumer through the same boundary | Accepted |
Anonymous unsigned-trailer requests were not prohibited wholesale. If bucket policy explicitly permits anonymous writes, they can still follow the anonymous authorization path. The forbidden state is the mixture of query credentials with no verification of those credentials.
Implementation and verification
The fix performs presigned rejection and SigV4 verification at the reader entry in cmd/streaming-v4-unsigned.go, while removing the gates in the PutObject and multipart handlers that depended on header presence.
New tests cover forged query PUT, multipart, mixed authentication, and anonymous policy. The historical record also includes a vulnerable-parent write that succeeded, failure on the patched tree, and live-server before/after smoke tests showing that header-authenticated and valid anonymous flows continued to work.
The public fix is f444b6f37. The live exploit was not rerun while preparing this article.
Compatibility and residual risk
- Presigned/query unsigned-trailer is now explicitly unsupported, an intentional breaking change.
- Moving the fix into the reader significantly reduces the chance that a sibling handler omits the check again.
- Other streaming authentication modes still need their own audits; this reader fix does not establish that every SigV4 streaming combination is safe.
The shape of this fix matters as much as its payload: when several handlers share an authenticated data stream, authentication belongs to the reader rather than to optional checks in each caller.
3.12 - CVE-2026-42600: ReadMultiple Storage-REST Path Traversal
Status: Released
First containing release: RELEASE.2026-06-18T00-00-00Z
GitHub advisory: GHSA-xh8f-g2qw-gcm7
Affected scope: Distributed erasure only; cluster-root / internode JWT required
The msgpack body of /rmpl carried Bucket, Prefix, and Files. The old code joined those values into filesystem paths without a containment check. The initial fix implemented full preflight validation. Continued call-chain review then found that this API had had no production caller since 2024. The final solution changed from “retain and harden” to removing the route, handler, client, interface, and generated code.
Deleting roughly a thousand lines was a larger source diff than a local validation guard, but it left a smaller long-term attack surface.
Threat model
The vulnerable route was registered only in distributed erasure mode; single-node deployments were unaffected. An attacker needed an internode JWT derived from the root secret, control of a node, or the ability to intercept unencrypted traffic between nodes.
The dangerous fields were inside the msgpack body, not the URL or form data, so upper HTTP path middleware never saw them. xlStorage.ReadMultiple joined and read the resulting paths directly, allowing them to escape the drive root.
This was not an anonymous S3 vulnerability. It crossed the boundary from “cluster root or peer” to “any file readable by the node process.”
First design: retain the API and validate it completely
The initial patch in xlStorage.ReadMultiple:
- rejected absolute paths,
.and..segments, backslashes, Windows drive prefixes, and NUL bytes; - checked final containment across drive, volume, prefix, and file;
- preserved the historical contract for an empty Bucket and
.minio.sys/multipartwhere possible; - returned an error before any read or streaming began.
That design could close the known traversal, but review quickly exposed an early-return gap.
MaxResults exposed the danger of validate-as-you-use
The first patch validated each Files item inside the read loop. For Files=[good, bad] with MaxResults=1, the function returned after reading the first item and never validated the second.
It did not read the second malicious file, but it violated the intended invariant that the entire msgpack request must be valid before execution. Validation was therefore moved into a complete preflight pass, with path-length checks also completed before streaming.
The general rule is useful beyond this endpoint: when a request can return early or stream a partial response, checking an element immediately before use is not equivalent to validating the whole request.
Final decision: delete the API
Further call-chain audit established that:
- upstream removed the last production caller in September 2024;
- multipart had moved to
ReadParts; - the current tree had no in-tree production consumer;
- upstream’s final remediation also deleted
ReadMultiple.
The final change removed the route, handler, client wrapper, StorageAPI / xlStorage method, metric, datatype, and generated code. storageRESTVersion retained the existing compatibility strategy.
| Option | Short-term change | Long-term maintenance surface | Decision |
|---|---|---|---|
| Validate in place | Smaller diff and preserved endpoint | Permanently retains an unused, privileged file-reading API | Abandoned |
| Delete the API | Removes more interface and generated code | Minimizes attack and maintenance surface | Accepted |
Verification and release
The in-place validation phase ran focused tests for xlStorage, the storage-REST client, msgpack encode/decode, and path edge cases; adversarial review exposed the MaxResults flaw. The deletion phase checked the route, client, interface, generated surfaces, and absence of callers.
The public fix is 73ac524, released with SILO 2026-06-18. The post-deletion full suite was not rerun while preparing this article.
Compatibility and claim boundary
- The external S3 API is unchanged.
- Third-party implementations that privately called the internal
/rmplendpoint will stop working. - A mixed-version rolling upgrade may encounter a protocol mismatch, so cluster nodes should be kept on the same version during the upgrade.
- Removing this endpoint proves only that
ReadMultipleno longer exists; it does not establish that every internal node request carrying body paths has completed a containment audit.
This CVE reached the right final fix, but it also leaves an important distinction: closing one endpoint is not the same claim as closing an entire vulnerability class.
3.13 - Internode Path Containment Audit: Paying Off What CVE-2026-42600 Left Owing
Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work. Affected scope: Distributed erasure only; cluster-root / internode JWT required Prerequisite reading: CVE-2026-42600 · ReadMultiple
This article is now public. The primary fix shipped as recorded above; residual findings below are the investigation-time record, not a claim that publishing one fix closed the entire defect class.
The previous entry closed with this sentence:
Deleting the endpoint proves only that
ReadMultipleno longer exists. It cannot be extrapolated into a completed containment audit of every internode body path.
That was an IOU, written down in plain sight. This is the record of paying it — and a not-very-flattering construction log.
Conclusions first
- Twelve defects, all inherited from upstream. Verified by per-function md5 comparison: the fork’s diff against upstream on the affected files is pure deletion, zero added lines.
- This is not a new vulnerability. It is the remainder of CVE-2026-42600 — three more protocol surfaces under the same root cause.
- Our failure is not in the code. It is in the record: a point fix was written up as a closure.
- While fixing it we introduced four regressions of our own, every one of them in a rule we invented rather than reused.
“N endpoints” was the wrong frame
Earlier audits kept counting endpoints, arriving at 19, then 21, then 22 — and missing an entire protocol surface each time. The real structure is four surfaces:
| Protocol surface | Entries | Covered by the global HTTP middleware |
|---|---|---|
| storage-REST HTTP query arguments | 9 | Yes — previously misreported as unprotected |
| storage-REST HTTP msgpack body | 4 | No; r.Form never comes from a body |
| storage Grid RPC | 18 | No; after one upgrade, frames never re-enter the HTTP chain |
| peer-S3 Grid RPC | 5 | No, and it bypasses getStorage() to reach drives directly |
The first row matters as much as the rest: it overturns the earlier “all 21 endpoints escapable” claim. Those audits grepped for the validation helper inside handler bodies, found nothing, and concluded there was no protection — missing that the protection lives in the middleware layer.
The fourth row is the one no amount of hardening in storage-REST handlers can reach.
The root cause is three layers, not one bug
Three design facts, none wrong on its own:
- Validation happens only at the HTTP surface.
r.Formis populated fromurl.ParseQuery(RawQuery)and never from a body (introduced 2017). - Grid RPC bypasses the middleware.
/minio/grid/v1upgrades once; subsequent msgpack frames never re-enter the HTTP chain (introduced 2023). - The storage layer performs no containment.
getVolDirrejects a volume only when it is exactly""/./.., andpathJoinrunsClean(settled 2018).
In one sentence: the upper layer assumes the lower one validates, the lower assumes the upper already did, and neither can see the channel in between.
ReadMultiple was merely one endpoint that exercised that structure. Removing it left the structure intact.
One line of the timeline deserves singling out. The divide-by-zero in ShardFileSize has been present since 2020, but only when it moved inside xioutil.WithDeadline in 2024-10 — a change meant to fix large-object timeouts — did it escalate from “one failed request” to “the whole process exits”, because WithDeadline runs its work function on a bare goroutine that no recover() can reach. That escalation was not visible at the time.
Vectors, confirmed by execution
Every one reproduced against a real xlStorage through the real REST/grid client, with planted sentinel files. None of this is static inference.
| Vector | Surface | Observed |
|---|---|---|
WriteAll("vol","../../x") |
storage grid | arbitrary file write outside the drive root |
RenameFile(".minio.sys","","bucket","x") |
storage grid | the entire system volume (IAM, config) relocated into a readable bucket, with no .. anywhere |
DeleteBucket("../victim", force) |
peer-S3 grid | recursive deletion of a tree outside the drive root |
DeleteBulk("vol","") |
HTTP body | whole volume moved to trash |
ReadAll(volume:"../") |
any | getVolDir’s check defeated by a trailing slash |
CheckParts with a zero Erasure |
storage grid | process terminates |
AppendFile declaring Content-Length: 64 GiB |
HTTP | 68,719,574,840 bytes allocated for an empty body |
DeleteVersions declaring 100M entries |
HTTP | a ten-byte parameter reserved 10.4 GB |
part Size = -2 |
storage grid | a truncated shard reported healthy; heal silently skipped |
Two of these had never been found before and are worth calling out.
RenameFile with an empty source path hits the volume-root alias and relocates the whole volume. Aimed at .minio.sys, one ordinary S3 GET afterwards yields the cluster’s IAM and configuration. It requires no traversal sequence at all — so any audit that greps for .. misses it by construction.
A negative part size floors both terms of ShardFileSize to zero. checkPart’s only integrity test is st.Size() < expectedSize, so every file that exists is reported intact, including a truncated shard. Worse, this holds whether or not the erasure parameters are valid: metadata that passes FileInfo.IsValid() — the very check healing trusts — is affected. That is not an input-validation problem but a data-integrity one: a legitimate heal reading poisoned metadata concludes the shard is fine and skips the repair.
The fix: two chokepoints, not twenty patches
The invariant to restore is one sentence: a path from an internode payload must resolve inside the volume it names, and a volume must resolve inside the drive root.
Only .. can break the first half (absolute and backslash-prefixed paths are folded under volumeDir by pathJoin’s Clean), and the second half has one independent break: paths that alias the volume root. Two rules, therefore — not a policy matrix.
| Chokepoint | Location | Coverage |
|---|---|---|
| Volume axis | getVolDir (4 lines) |
every caller, including peer-S3 |
| Path axis | decorator at getStorage() |
31 remote entries plus nested fields |
Under 40 lines of core logic. No handler is modified, no call site in xl-storage.go is touched, and the local erasure path is left alone.
Two details worth recording:
- The check must run before the join, on the raw argument.
pathJoinrunsCleanagainst an absolutedrivePath, which erases a leading..entirely —/drive/../../etcbecomes/etc, so a check placed after the join reads clean and passes everything. This is the most likely way a future refactor silently undoes the fix. NSScanneris the one method that reaches the filesystem withoutgetVolDir. Its guard line is load-bearing, not decorative.
Among the rejected alternatives, the notable one is adding containment at all 33 pathJoin(volumeDir, …) sinks. That would be genuine defence in depth, but it means 33 edits in the most performance-sensitive file in the tree, each needing its own judgement about whether the volume root is a legitimate target. The guard rails buy most of the same resistance to drift for a fraction of the risk. This is an explicitly recorded IOU: if a code path is ever added that reaches the filesystem without getVolDir, the decision must be revisited.
Construction log: four regressions we caused ourselves
This section is unflattering and more informative than the fix.
First: whitespace. The initial rule treated whitespace as a separator, refusing " " and " " — legal S3 object keys. PutObject commits through RenameData, so such a key would fail on every remote drive simultaneously and break write quorum. The irony: the vulnerability needs root credentials; this bug needs a user to send a space.
Second: backslash-only keys. Same function, same root cause. path.Clean never treats \ as a separator, so on Unix "\\" is an ordinary filename. Refusing it made a distributed cluster reject a write a single-node server accepts — the same S3 API behaving differently by deployment topology.
Third: spaces and periods on Windows. The Win32 normalisation layer strips trailing spaces and periods from a path component, so a component made only of those vanishes and the path resolves to its parent. That makes both " " and "..." volume-root aliases on Windows — and "..." was sitting in our own list of legal object names at the time. We had not merely missed the vector; we had asserted it was safe.
Fourth: negative part sizes. The new guard rejected only “positive size with unusable parameters”, equating “non-zero” with “positive”. Negative values take a different route to the same zero.
Two false greens
Writing the AppendFile acceptance test produced two meaningless green runs in a row:
- Driving it through the REST client — which special-cases
*bytes.Readerand derives Content-Length from it, silently overriding the forged value. - Switching to an opaque reader — at which point Go’s own HTTP client refuses to send a request whose body is shorter than the declared length.
Only driving the handler directly through httptest reproduced it. The lesson: a client’s self-protection is not a server’s defence, and an attacker with a raw socket has no such scruples.
A third was a design failure. We built a per-field reflection poisoner, then discarded it: it cannot distinguish “should have rejected but delegated” from “correctly allowed a non-path field” (ETag, Algorithm, …), so it reports correct behaviour as failure.
The subtlest lived in the fuzzer. The first property test treated separator-only strings as an exception with an early return. That is not an exception, it is a blind spot — the fuzzer had been shut out of the entire category by hand and could never have found the backslash key in a million executions. A wrong exception is more dangerous than no fuzzer at all, because it creates the impression the space has been searched.
A very concentrated pattern
| Component | Where its semantics came from | Regressions |
|---|---|---|
guardPaths |
reused existing hasBadPathComponent |
0 |
getVolDir guard |
reused existing hasBadPathComponent |
0 |
isVolumeRootAlias |
invented | 3 |
guardErasureParams |
invented | 1 |
The reused semantics produced zero regressions; the invented rules produced all of them.
This is not coincidence. hasBadPathComponent is already the object layer’s own rule via IsValidObjectPrefix, validated by real S3 traffic for years, and structurally cannot reject anything creatable through the S3 API. An invented rule has nothing behind it but the author’s imagination.
The actionable form: reuse rather than invent; and when you must invent, write the property test by exclusion rather than enumeration, express exceptions with a predicate independent of the implementation, and keep them as few as possible.
The guard rails matter more than the patch
The final test suite pulls in two directions, and neither alone is enough:
- Falsifiability — remove each guard in turn and confirm the tests actually go red (191 failing subtests with the traversal guards removed; 64 GiB and 10.4 GB reappearing with the allocation guards removed). This is precisely what the rejected community PR lacked: its test asserted
err != nilagainst a target that did not exist, so it passes with the vulnerability fully intact. - Legal-traffic fuzzing — asserting that any key
IsValidObjectNameaccepts, the guards accept (1.96M executions, no violations), and that any legal bucket name survivesgetVolDir(810K). This is what our own first two attempts lacked.
Plus a method-level reflection rail that fails by name when a path-taking method is added to StorageAPI unguarded.
History states the case for these rails bluntly: CVE-2026-39414 was also point-fixed on 2026-04-15 and only received a fix: complete ... two months later. Counting this one, “point fix → recorded as closure → completed months later” has now happened twice in this fork. The problem is not that someone was careless. It is that nothing in the tree could tell you a class was still open. Guard rails turn “someone must remember” into “CI fails”.
On adversarial review
This fix went through five rounds of independent adversarial review. Each round found one missed defect, and all five stood: whitespace keys → backslash keys → the AppendFile allocation → Windows spaces and periods → negative part sizes.
Our own review did find two in the same period (the WithDeadline log amplification and ReadParts using the wrong rule), but only after being pushed to that standard.
The hit rate says something plain: the last gate before merge should be independent acceptance, not the author’s own conclusion. During this work the author judged the change ready to ship four times and was overturned three.
Follow-up status
The following five follow-ups were not published at the original August 3 review point. They subsequently shipped in Server 20260804:
ReadFileHandleris bounded. Commitb6f70ab08rejects a declared read length above 5 GiB, the maximum size of the S3 part represented by this legacy whole-file bitrot path. Legitimate GiB-scale reads can still allocate on that scale; the change removes caller-controlled allocation above the format’s real ceiling rather than pretending large reads are cheap.- Negative part sizes cannot be persisted or trusted. Commit
80e8eaa42rejects them at theAddVersionwrite funnel and again inCheckPartsandVerifyFile, so both new poison and already-written metadata are covered. The internode boundary check uses the same predicate. - Non-positive erasure block sizes are rejected at construction. Commit
80e8eaa42validatesblockSizeinNewErasure, covering the other offset and decode divisions that a single downstreamShardFileSizeguard could not. Rebalance’s separate division is guarded at its own boundary. ReadPartserrors finally reach the caller (1af351a70). A pre-existing bug, fixed in passing: the handler declared success (done(nil)) before the storage call finished, so everyReadPartsfailure — not only the new guards, buterrFileNotFoundand friends too — was never returned; the client received a keep-alive stream and decoded the error text as msgpack, surfacing asmsgp: attempted to decode type "int" with method for "map". After the fix, real storage errors reach quorum reduction; success paths are untouched, and that msgpack decode noise in multipart-related logs is replaced by actual storage errors.- A recovered panic is not a healthy node (
a36fd8fff). TheWithDeadlinerecovery turns what used to be an instant process death into a stderr line (panic in deadline-bounded work: ...with a stack) plus one failed request. Stack dumps are capped at ten per process lifetime, but the error returns every time. If your alerting only watches process liveness, add a log alert for that string — otherwise a real bug that previously announced itself as a crash now only whispers in the logs.
Two limitations remain and should not be folded into a stronger claim:
- No Windows CI. Windows builds are published; tests run on Ubuntu only. The Windows rule is reasoned from documented Win32 behaviour and has not been verified on the platform.
- Symlinks. The containment check is lexical, as upstream’s is.
Closing
The previous entry said that closing an endpoint and closing a defect class are two different conclusions. This time the known sinks are closed on the local branch, at the cost of four self-inflicted regressions and three overturned declarations that it was ready to ship. At the investigation date publication remained a separate gate; these fixes subsequently shipped in 20260804.
If only one sentence survives: the vulnerability was upstream’s; our mistake was treating a point fix as a closure. And what prevents a third occurrence is not a more careful person — it is a test that fails.
3.14 - The Parser Knew, the Schema Didn't: Config Keys That Could Take Every Notification Down
Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work.
Classification: Configuration-schema consistency and availability, not a vulnerability; includes one defensive hardening (credential values no longer echoed in validation errors)
Affected scope: notify_nats JWT/NKey/TLS-handshake-first options, notify_amqp immediate, and any pre-2020 config migrated with an enabled NATS target — whose failure then silences every notification backend
Tracking: pgsty/minio issue #39
This article names two unfixed availability defects in neighbouring code (the Postgres/MySQL migration writes, and
kvFieldstypo folding). Neither is exploitable — both break the operator’s own configuration, loudly or not at all — and both are already named in the committed audit test’s allowlist. Publication needs no hold beyond the release itself.
Conclusions first
- Three
notify_natsoptions —user_credentials,nkey_seed,tls_handshake_first— and onenotify_amqpoption —immediate— were read by the parser, written by the legacy migration, and registered nowhere.CheckValidKeysrejected exactly whatGetNotifyNATSrequired. - One constant meant two things.
target.NATSUserCredentialsheld the string"MINIO_NOTIFY_NATS_USER_CREDENTIALS", sat in the environment-variable const block, and was used both as an env var name and as a config key. The snake_case config key for creds-file auth did not exist anywhere in the program. - The reporter’s error did not come from their command. It came from the legacy migration: the pre-fix migration reproduces the issue’s error text byte for byte, including the
notify_nats:ONEtarget name their command never mentioned. The migration wrote the store once; validation rejects it at every boot thereafter. - The blast radius is the amplifier:
FetchEnabledTargetsfails fast on the first bad subsystem, its only caller just logs, and the global target list staysnil— so one broken NATS entry silently switches off Kafka, webhook, MQTT, and everything else. - Inherited from upstream. Three feature PRs — #19139 (2024-02,
user_credentials), #21008 (2025-04,tls_handshake_first), #21231 (2025-04,nkey_seed) — each added the parser and the env var, and each skipped the schema. Upstream is archived; the fork inherits both the defect and the duty. - The fix registers the keys, splits the two-faced constant, corrects the migration — including a sibling bug that silently wrote
immediate’s value under theinternalkey — tolerates the legacy on-disk spelling on the load path only, stops echoing values in invalid-key errors in bothCheckValidKeysforms, and installs an AST audit that mechanically forbids this defect class across all ten notify subsystems. - The audit found the next instance before the ink dried: the Postgres/MySQL legacy migrations write five unregistered keys, one of which is a plaintext database password. Recorded, allowlisted shrink-only, tracked for follow-up.
The error that named a target nobody asked about
The report (issue #39, by kuldeep-link11, against a NATS cluster using JWT operator/accounts auth) is a clean reproduction: configure notify_nats with a credentials file, watch it bounce.
Two things in that error are wrong in ways the command cannot explain. The invalid-key list contains nkey_seed= and tls_handshake_first=off — keys the user never passed. And the rejected sub-system is notify_nats:ONE, while the command configured notify_nats:FITCHECK.
The second oddity is the whole case. Our reviewer established that the mc admin config set path cannot even carry an unregistered key: the server-side tokenizer, kvFields, splits the input line by searching for registered key names, so an unknown token never becomes a key at all — it is absorbed into the preceding key’s value. Probed directly:
So the rejection could not have been about the command line. It was validateConfig sweeping the whole subsystem and tripping over a different, already-stored target named ONE that carried all three keys. Only one code path in the tree writes those key names into a store: the legacy config migration. Driving the pre-fix migration on an enabled NATS target named ONE reproduces the issue’s error text character for character — including the empty nkey_seed=, which is just what migration writes when the legacy config had no NKey.
That reframes the incident. This was not “the server rejected my command.” It was: an old config was migrated once, the migration wrote three keys the validator does not accept, and the store has been failing validation at every boot since — taking every other notification target down with it, silently, because the failure is logged and swallowed. The reporter’s command merely walked into the blast radius and got handed someone else’s error.
One constant, two meanings
The declaration, as inherited (internal/event/target/nats.go, pre-fix):
NATSUserCredentials is named like a config key, valued like an env var, and shelved with the env vars. The parser used it as both: once as the env var to look up, once as the config key to read from the stored KVS. The migration used it as a key to write. There was no "user_credentials" string anywhere in the program — the config key for creds-file auth simply did not exist, which is why the reporter, finding no documented key, resorted to passing the env var name as one.
A name that means two things will eventually be wrong in one of them. Here it was wrong in both directions at once: as a key it was unregistered garbage; as the only spelling available it taught users and the migration to write garbage.
Four surfaces, no handshake
A notify option in this codebase lives on four surfaces that must agree: the defaults (DefaultNATSKVS — what validation accepts and mc admin config get displays), the help (HelpNATS — what mc admin config documents), the parser (GetNotifyNATS — what the server actually reads), and the migration (SetNotifyNATS — what upgrades write). Nothing ties them together. Three upstream feature PRs each updated the parser and the env plumbing, and each forgot the first two surfaces:
| Key | Parser reads | Migration writes | Defaults | Help | Introduced |
|---|---|---|---|---|---|
user_credentials |
yes (via the two-faced constant) | yes (as the env-name string) | no | no | #19139, 2024-02 |
nkey_seed |
yes | yes | no | no | #21231, 2025-04 |
tls_handshake_first |
yes | yes | no | no | #21008, 2025-04 |
immediate (AMQP) |
yes | see below | no | no | config-KV rewrite era |
The AMQP row hides the quieter sibling. The AMQP migration did not skip immediate — it wrote immediate’s value under the internal key, and dropped cfg.Internal entirely:
Because internal is registered, this one passes validation. The NATS gaps break a migrated config loudly enough to be found eventually; the AMQP gap corrupts it silently — a migrated broker config carries the wrong flag with a clean bill of health. One defect class, two presentations: the unregistered key fails closed, the misrouted value fails wrong.
The amplifier
None of this would deserve the word “outage” without the aggregation semantics. FetchEnabledTargets iterates the ten notify subsystems and returns (nil, err) on the first failure; its only caller logs the error and moves on, leaving the global notification target list nil; every later lookup nil-guards into an empty list. One rejected notify_nats target therefore turns off all bucket notifications — Kafka, webhook, AMQP, MQTT, the lot — with nothing but one line in the server log.
We considered changing this to per-subsystem isolation and decided not to, in this fix. Skip-the-broken-subsystem is a real behavioural change to how operators experience a bad config: today it fails loudly-in-aggregate (everything stops), and configurations that operators have already reasoned about depend on validation being all-or-nothing. Rewiring that is a compatibility decision that deserves its own change, not a rider on a registration fix — and once registration is correct, legal configs no longer trigger the cascade at all. The decision is recorded as a doc comment on FetchEnabledTargets and pinned by a characterization test, so the next person to touch it changes it on purpose or not at all.
The fix
About a hundred lines of production change, carried by nine hundred lines of tests (162ded343: 8 files, +1029/−7).
Registration. All four keys enter their default KVS and help schema, placed where an operator would look for them (user_credentials beside username, nkey_seed after token, tls_handshake_first after tls_skip_verify, immediate beside mandatory). Registration is also what makes a key visible: all four now appear in mc admin config get output where they previously did not.
The constant, split. NATSUserCredentials becomes a real config key, "user_credentials"; a new EnvNATSUserCredentials carries the env string. Every env var name involved — MINIO_NOTIFY_NATS_USER_CREDENTIALS, _NKEY_SEED, _TLS_HANDSHAKE_FIRST, MINIO_NOTIFY_AMQP_IMMEDIATE, and their _TARGET-suffixed forms — is frozen byte-for-byte: they are public interface, they worked throughout (the env route was always the workaround), and a test now pins them as raw string literals, so no rename of a Go constant can drift them silently.
Help flags, by precedent. Both new NATS values are file paths (a .creds file; an NKey seed file), so they are marked Sensitive but not Secret, mirroring cert_authority/client_cert/client_key rather than password/token. Secret would additionally redact them from mc admin config get — hiding an operator’s own configured path from them, which is why the private-key path client_key never had it either.
Migration, corrected. SetNotifyNATS now writes the real key; SetNotifyAMQP writes immediate = cfg.Immediate and internal = cfg.Internal.
If you are affected today, on a pre-fix build: the env var route works and always did, and mc admin config reset myminio notify_nats:<target> un-wedges a poisoned store at the cost of its settings. On the fixed build, poisoned stores simply load again — next section.
Living with what the old migration already wrote
Fixing the migration helps the next upgrade. It does nothing for stores the broken migration already wrote, which contain the literal key MINIO_NOTIFY_NATS_USER_CREDENTIALS — still unregistered, still fatal at every boot. Telling those operators to hand-reset their config would mean punishing them for our write.
So the load path tolerates it, narrowly. Validation accepts the legacy spelling for the NATS subsystem only — a test asserts AMQP still rejects it, so the tolerance cannot become a general escape hatch — and the parser falls back to it only when the real key is empty. Precedence is env > user_credentials > legacy key, and it holds by construction rather than by convention: the fallback result is passed as the default argument of the env lookup. All three orderings are tested. The legacy key stays out of the defaults and the help on purpose: it is tolerated, never advertised, never newly settable (kvFields sees to that).
The constant for it is a package-local literal, not an alias of EnvNATSUserCredentials — deliberately. It names bytes already on disk, so it must not follow any future rename of the env constant. The comment says so.
One trap discovered while wiring this, worth its own paragraph because it will bite someone eventually: the codebase has two CheckValidKeys — a free function and a method — and their deprecatedKeys parameters mean opposite things. The free function tolerates the listed keys (skips them); the method subtracts them from the valid set (rejects them). Refactoring this call from one form to the other would silently invert the tolerance into a ban. That asymmetry is now documented at the call site, which is the best one can do short of renaming an exported API.
The tolerance is written to be retired: the clean end state is to rewrite the legacy key into user_credentials once at load, then delete both the tolerance and the fallback. That is follow-up #2 below — and it also closes a small hole the tolerance leaves open: an unregistered key carries no Sensitive flag, so a tolerated legacy key ships its value (a path) unredacted in health-diagnostics bundles while user_credentials shows *redacted*.
Secrets in error messages
The invalid-keys error that started all this printed the rejected pairs with their values: found invalid keys (MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/creds/minio_notifier.creds ...). Those paths are mild. The mechanism is not: whatever value rides on a rejected key — a mistyped nkey_sed=<seed>, a bind password on a stale LDAP key — lands in the server log and in the mc client’s terminal.
Both CheckValidKeys forms now print key names only, keeping the shape and the mc admin config reset hint. The second site was one step beyond the written task scope — the method form serves LDAP, OpenID, and the policy plugin, where a rejected value can be an actual bind password — and the independent review, asked to judge that extension, said it would have demanded it: fixing one of two identical leaks is a half-fix. Repo-wide, nothing parsed values out of that string and no test asserted the old text; the change is global and intended.
There is a converse worth recording: this redaction is what stands between the next defect of this class and a credential in the logs. The follow-up below found the Postgres/MySQL migrations writing a plaintext database password under an unregistered key — on a pre-redaction build, the resulting rejection prints that password.
A guard that makes the class extinct
Registering four keys fixes four keys. The class — four surfaces, no handshake — stays open unless something ties the surfaces together mechanically. The fix therefore ships an AST-based audit test that parses parse.go and legacy.go, resolves the constants (from the target package sources, so there is no hand-maintained list to rot), and asserts, for all ten notify subsystems:
- every key the parser reads is registered in that subsystem’s defaults;
- every key the migration writes is registered (minus an explicit, shrink-only allowlist — next section);
- every help entry names a registered key.
Run against the pre-fix tree, it fails on exactly the four known gaps and nothing else — which is the red proof that it measures the right thing.
The adversarial review then attacked the audit itself with a mutation harness, on the theory that a guard you cannot watch fail is a guess — the discipline the previous article argued for. Seven of its nine mutations were caught. Two were not, and both blinded the audit silently: rename the parser’s loop variable (the read-collector pattern-matched the receiver name kv), or switch a migration entry to Go’s idiomatic elided composite-literal form (the write-collector demanded a typed config.KV{...}). In both cases the collector returns an empty map, the assertion loop iterates zero keys, and the test passes vacuously. Both are refactors a maintainer would make without a second thought; one of them is what gofumpt nudges you toward.
Two hardenings closed this, each verified in both directions — with the hardening the mutation is caught; with the hardening removed (the counterfactual) the vacuous pass returns:
- A floor assertion in the reverse direction: every registered key must be seen being read. This holds for all ten subsystems today — measured, not assumed, including the deprecated
streaming_*keys read inside a nested conditional — so it costs nothing, and a blinded collector now produces one loud error per registered key (22 of them for NATS) instead of a green run. - A widened literal guard: typed literals that are neither
config.KVnorconfig.KVSare skipped; untyped (elided) literals have no type to check and are now inspected rather than ignored.
Final score: ten mutations, ten caught — the harness gained one variant along the way, and the closure round swept the full suite. The audit also enforces its own allowlist in both directions — removing an entry that is still needed fails, and an entry that goes stale (the migration no longer writes that key) fails too, so the allowlist can neither grow silently nor lie about the present.
What the audit found next
The write-side check refused to go green on two subsystems that had nothing to do with issue #39. SetNotifyPostgres and SetNotifyMySQL write five keys — host, port, username, password, database — that no default KVS registers and no parser reads. These are relics of the pre-DSN configuration shape, and the migration still emits them. Driving the real helpers confirms it: a migrated Postgres or MySQL notify target is rejected on the next load with found invalid keys (host, port, username, password, database) — the same failure mode as NATS, the same every-boot persistence, the same all-notifications blast radius through the fail-fast. And password there is a plaintext database password, which is exactly the value the redaction above now keeps out of the logs.
It is deliberately not fixed in this change. The scope was locked to the NATS and AMQP gaps, and the right treatment (register the five as deprecated, or stop writing them, or both) is a judgment call that deserves its own red/green cycle. It is pinned in the audit’s knownUnregisteredWrites allowlist with a shrink-only comment, so it cannot be quietly forgotten: the day someone fixes it, the stale allowlist entry fails the test and demands its own deletion.
Open items, none in a released build as of 2026-08-04:
- Postgres/MySQL migration unregistered writes — major, live at every boot for anyone migrating a pre-KV config with those targets enabled.
- Rewrite-on-load for the legacy NATS key, then retire the F5 tolerance and fallback; also closes the health-bundle redaction gap for tolerated keys.
kvFieldstypo folding — an unknown key name inmc admin config setis silently absorbed into the preceding key’s value instead of erroring. Pre-existing upstream wart; it protected nobody here and will corrupt someone’ssubjecteventually.
Review record
The change went through three gates before commit:
| Gate | Method | Outcome |
|---|---|---|
| Implementer | tests written first and run against the unmodified tree; the missing constant made the suite fail to compile, which is itself the red for the split; targeted reversal produced runtime reds for the rest | red established for every claim |
| Independent adversarial reviewer | detached worktree at the pre-fix commit; re-derived every red rather than trusting the report; mutation harness against the audit; probe tests for precedence edges; reproduced the reporter’s error byte-for-byte from the migration path | REVISE, two demands |
| Closure round | both demands applied; counterfactual mutation runs (with and without each hardening) prove the hardenings load-bearing; reviewer re-diffed, re-ran, re-mutated | ACCEPT, 10/10 |
The honest accounting, in the house tradition: neither demand was a defect in the production fix. One was a lint gate (two British spellings that would have failed make test — and while fixing them, the implementer’s rewritten comment introduced a third, dialled, which the same gate caught; the demand vindicated itself in real time). The other was the audit-blindness pair above — durability of the guard, not correctness of the change. What the review did overturn was the incident’s origin story: the migration-path reproduction, the ONE target, and the kvFields absorption proof all came from the reviewer, and they change what operators should conclude — this was a boot-time outage lying in wait in stored configs, not a CLI validation quirk.
The implementer’s red phase also surfaced three bugs in its own new tests before the fix landed, recorded rather than smoothed over: a fixture that assumed stored targets are layered over defaults when config.Merge actually passes them through verbatim; a characterization test that segfaulted on a nil HTTP transport (FetchEnabledTargets dereferences it unconditionally — hostile to testing, noted, unfixed); and an early draft keyed to the very constant the fix renames, which made it pass green pre-fix — rewritten against the literal string so it pins the on-disk schema rather than the Go symbol.
Declined, and left open
Declined, deliberately:
- Per-subsystem error isolation in
FetchEnabledTargets— a compatibility decision, not a rider (above). - Registering or advertising the legacy key — tolerated on load, absent from defaults and help, impossible to set anew.
- Renaming
EnvNatsTLSHandshakeFirst’s odd casing — an aesthetic rename in a fork is diff noise that buys nothing. - Fixing the Postgres/MySQL migration here — scope-locked, pinned in the allowlist instead (above).
Left open: the three follow-ups above, and one cosmetic consequence — a store that still carries the tolerated legacy key will show it verbatim in mc admin config get until rewrite-on-load lands.
Closing
Every one of these keys worked perfectly through the environment variable, which is why three feature PRs could ship, get reviewed, get used, and never notice that the config-file half of the interface was stillborn. The parser and the schema are two descriptions of the same contract, maintained by hand, four surfaces wide — and for two and a half years nothing in the build checked that they agree.
If only one sentence survives: when two artifacts must stay identical and only convention binds them, the divergence is not a risk but a schedule — put a machine between them, then mutate the machine until you have watched it catch the drift you fear.
Follow-up status: the Postgres/MySQL discrete-field migration left open at this review point was repaired by f1ba68358 and shipped in Server 20260903. Enabled targets without a canonical DSN now explicitly prevent startup. See database notification migration; this does not close the separate tokenizer observations.
3.15 - Object Grant, Bucket Reach: When 'bucket/*' Could Rewrite the Bucket Itself
Status: Fixed on pgsty/silo-pkg main (3c24ad1, extended by 1f97549, scoped to its final twelve actions in d8b1fa7), released as silo-pkg v3.11.0; consumed by pgsty/minio
Classification: Access-control hardening — a privilege boundary, narrowly restored
Affected scope: IAM users/roles/service accounts granted only object-scoped (arn:aws:s3:::bucket/*) access, in deployments that share a cluster across tenants
Tracking: upstream minio/minio issue #20449 (public since 2024, still open)
Conclusions first
- In IAM policy matching, a bucket-level request carries an empty object name, and the matcher built its resource string as
"bucket/". An object-only policy pattern —"arn:aws:s3:::bucket/*"— then matched that string, so a grant that should cover only objects also authorized bucket-level actions. - The dangerous one is
PutBucketPolicy. A tenant holding onlys3:*onbucket/*could install a bucket policy withPrincipal:"*"— making the bucket publicly readable or writable — or grant itself bucket-level control. Same mechanism, same class:DeleteBucket/ForceDeleteBucket(the issue’s own reproduction),PutReplicationConfiguration(exfiltration),PutBucketLifecycle(mass deletion),PutBucketVersioning,PutBucketObjectLockConfiguration, and the rest of the bucket-configuration writes. - The full correction is a two-directional behavior change: it tightens over-granting
Allowstatements and loosens over-blockingDenystatements, and it would revokeListBucket/GetBucketLocationgrants that many real deployments write asbucket/*today. That is a compatibility break, not a clean patch. - So we shipped a narrow fix — first six sensitive bucket-configuration writes, then, in a second pass, twelve: the bucket-level writes that hand the caller something its object access does not already give it, plus four that no handler implements. Only on
Allowstatements, so noDenyand noNotResourceexclusion is ever weakened, with an environment-variable escape hatch. The compatibility-sensitive read/list family,CreateBucket, and three bucket writes with plausible tenant use are left unchanged, by decision. - Twice we claimed the change could only remove permissions, and twice an untested case said otherwise — the second time found by an independent review of a shipped release. The protected path now requires the resource to match both the bare and the historical form, which makes the property hold by construction rather than by argument.
- The fix is red/green proven at the matcher layer and end to end through the real handlers; the object-scoped hot path is untouched.
The slash, and the empty object name
Every bucket-level S3 operation authorizes with an empty object name — checkRequestAuthType(ctx, r, policy.PutBucketPolicyAction, bucket, ""). The IAM matcher turned that into a resource string, and for the empty-object case it appended a trailing slash:
"bucket/" is matched by the wildcard pattern "bucket/*", because * matches the empty string. So a policy that grants s3:* on arn:aws:s3:::bucket/* — which reads as “anything, but only on the objects in bucket” — was evaluated as granting bucket-level actions too. The bucket-policy evaluation path (for anonymous/public access) never had this slash and is the reference-correct behavior; only the IAM path was wrong, and there was exactly one place it went wrong.
This is upstream minio/minio #20449, filed in 2024. An early upstream attempt deleted the slash outright and was reverted the same day for breaking policies that relied on the old behavior. The lesson we took from that revert shaped the fix below.
What it actually enables
PutBucketPolicyHandler has a single authorization gate and nothing behind it. Once the IAM check passes, the caller may store any well-formed bucket policy for that bucket.
The concrete chain, in a multi-tenant cluster:
- An administrator grants tenant A the policy
Allow s3:* on arn:aws:s3:::bucket-a/*, intending “A may work with the objects inbucket-a, nothing more.” - Because of the slash, A may call
PutBucketPolicyonbucket-a. - A installs
{ "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::bucket-a/*" }. Every object inbucket-ais now readable by the anonymous internet.s3:*makes it world-writable. PointingPrincipalat an account A controls exfiltrates the data; granting itself bucket-level actions in that policy is self-escalation.
The same object-only grant reaches other bucket-configuration writes with comparable consequences: replication to an attacker’s target, a one-day lifecycle expiry that deletes the bucket’s contents, disabling versioning, tampering with object-lock retention. None of these should be reachable from a grant scoped to objects.
This is not remotely exploitable and requires no missing credential — the caller is an authenticated principal you deliberately gave a scoped policy to. In a single-tenant deployment, that principal is your own trusted user and the practical risk is low. In a shared, multi-tenant cluster it is a real cross-tenant boundary failure.
Why a narrow fix, not the whole boundary
The obvious fix is to stop appending the slash for every bucket-level request. We did not do that, for two reasons that matter more than the one-line diff suggests.
It breaks common, benign usage. The correction does not only revoke the dangerous bucket writes — it also revokes ListBucket, GetBucketLocation, and ListBucketMultipartUploads when they were granted through bucket/*. Many deployments write exactly that and rely on it. The evidence is upstream’s own test suite: eleven STS integration tests grant s3:ListBucket on bucket/* and then assert that listing works. If the projects that wrote the server write it this way, production policies do too. A maintenance upgrade that turns those into AccessDenied is precisely the kind of surprise we refuse to ship.
It cuts both directions. The matcher builds the same resource string for Allow and Deny. So the full correction tightens over-granting Allow statements and simultaneously loosens over-blocking Deny statements: an administrator who locked a bucket with Deny s3:* on bucket/* would silently lose that protection for bucket-level actions. A clean-looking fix that moves security in two directions at once is not a maintenance patch — it is a migration.
So we narrowed the change to where it is unambiguously right and effectively free of compatibility cost:
- Only bucket-level writes are protected. The first pass covered six sensitive configuration writes:
PutBucketPolicy,DeleteBucketPolicy,PutReplicationConfiguration,PutBucketLifecycle,PutBucketVersioning,PutBucketObjectLockConfiguration. The second pass (below) extended that to twelve. Almost nobody grants these through an object-only pattern on purpose — you do not accidentally rely on an object grant being able to rewrite a bucket’s policy or delete the bucket — so revoking that path breaks essentially no one. - Only on
Allowstatements.Denystatements keep the historical resource string, so no existingDenyis ever weakened. The narrow fix only ever adds a denial. - The read/list family is left exactly as it was.
ListBucketonbucket/*still works. That is the compatibility-sensitive part, and it waits.
The fix
The matcher keeps the trailing slash in every case except one: a bucket-level Allow statement being evaluated for a protected action, with the compatibility shim off.
Because args.Action is the concrete request action, a wildcard grant (s3:*) is covered too: the wildcard matches at the action step, and by the time the resource string is built the action is the specific PutBucketPolicy. A bare-bucket resource (arn:aws:s3:::bucket) and the * resource still match, so correctly scoped grants — including the built-in readwrite policy — are untouched.
The escape hatch is MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on, read once at startup. It restores the full historical behavior — both the over-grant and the over-block — for any operator who needs the old semantics while they adjust their policies.
The second pass, and the question that decided its size
The first round protected six configuration writes and stopped. Reviewing it against the original issue showed that was not enough: the action reproduced in #20449 itself — DeleteBucket — was still reachable through an object-only grant. An end-to-end test against the first-pass build confirmed it: a user holding nothing but s3:* on arn:aws:s3:::bucket/* called RemoveBucket and the bucket was gone.
Extending the set raised the real question — how far? The first instinct was “every bucket-only write except CreateBucket,” fifteen actions. That was the wrong instinct, and the reason is a detail of how the bug fires.
The bug only triggers when the statement already grants the bucket-level action. Resource matching runs after action matching, so a read-only tenant holding s3:GetObject on bucket/* never reaches DeleteBucket — the action never matched. In practice the affected principal holds s3:*, which means they already have full read, write, and delete over every object in the bucket. That reframes the severity of each candidate action, because the question is not “how dangerous is this action in the abstract” but “what does reaching it add to a position that already includes all the data?”
By that test, three groups fall out:
Protected — reaching these grants something the object access does not. PutBucketPolicy and DeleteBucketPolicy hand access to other principals, anonymous included, and can grant the caller bucket-level actions it was never given: self-escalation and public exposure. PutBucketObjectLockConfiguration and PutBucketVersioning defeat protections that exist precisely to stop a holder of write access from destroying data. PutReplicationConfiguration and PutBucketLifecycle act under server credentials and keep acting after the caller’s access is revoked. DeleteBucket and ForceDeleteBucket destroy the bucket entity and its configuration irreversibly.
Protected at zero cost. PutBucketCors, DeleteBucketCors, PutBucketQOS, and PutInventoryConfiguration have no MinIO server behavior attached today — no handler at all, or a handler that returns NotImplemented after the authorization check. Withholding them changes nothing that works, and covers them in advance if a handler is ever wired.
Deliberately not protected. PutBucketTagging, PutBucketEncryption, and PutBucketNotification are bucket-level writes, and the first draft of this pass did protect them. They came back out. None of the three gives the caller access it does not already hold — the harm is to the owner’s posture, not to the access boundary — while a tenant handed s3:* on bucket/* and told “this bucket is yours” may quite reasonably tag it, set default encryption, or wire up event notifications. Low security gain against a real compatibility cost is the wrong trade for a maintenance release. They keep the historical matching, and a test now asserts that they are unprotected, so putting any of them back is a deliberate act with a visible cost rather than an edit to a list.
That leaves twelve actions, shipped as silo-pkg v3.11.0. Two older boundaries stand unchanged: CreateBucket keeps the historical matching (it targets a bucket that does not exist yet, and provisioning flows commonly create a tenant’s bucket with that tenant’s own credentials), and the read/list family still waits for the migration-gated change.
The choice of what to break, in other words, was made by asking would an administrator ever write this on purpose — not by ranking the actions by how dangerous they sound. The first question predicts which upgrades break; the second only sets urgency.
The claim that was wrong twice
Everything above rests on one property: this change may remove permissions and must never add one. Both times we asserted it, we were asserting it about a mechanism we had reasoned through rather than tested through. Both times it was false.
The first pass withheld the slash from the NotResource match as well — and NotResource is an exclusion. An Allow s3:* NotResource bucket/* statement historically did not apply to bucket-level requests on that bucket; matching the exclusion against the bare bucket name made it stop matching, so the Allow it qualified grew, for exactly the writes being protected. Restoring the historical form for NotResource fixed that, and the second pass shipped saying the result was “provably monotone.”
An independent adversarial review of that release produced a counterexample within the hour. Withholding the slash does not merely remove a match — it changes which string patterns are matched against, and a pattern can match "mybucket" without ever having matched "mybucket/". The clean case is a fixed-width wildcard:
? matches exactly one character. Against the historical nine-character "mybucket/" it does not match, so this statement never authorized the bucket-level write. Against the new eight-character "mybucket" it matches, so the hardening granted something the buggy matcher refused. Small in reach — you have to write a length-sensitive pattern — but it is precisely the class of defect the property was supposed to exclude, shipped in a release whose notes claimed the property held.
The fix is not another special case. On the protected path the matcher now requires both forms to match: the bare bucket name and the historical "bucket/". The result is an intersection with the historical decision, so it is monotone by construction — there is no pattern it can newly satisfy, and no argument to get wrong next time. mybucket* still grants (it matched both all along); mybucket/* is still withheld; mybucke? is refused exactly as it always was. That shipped as silo-pkg v3.11.0.
Two things are worth taking from this beyond the patch itself. A correctness fix in an authorization path must never make anything newly allowed — and the only way to know is to test both directions, because the reasoning feels airtight in both cases where it wasn’t. And when a security property is load-bearing, build it out of an operation that cannot violate it rather than out of a case analysis you believe is complete.
Regression tests now pin each direction: grants narrowed, Deny untouched, NotResource exclusions untouched, fixed-width wildcards not broadened, the three unprotected writes still reachable, plus an invariant test that every protected action really is bucket-only (ResetBucketReplicationState, despite its name, is an object action and stays out). In the server they run end to end through the real handlers — client, inline session policy, and the S3 router — and every one of them fails against the release that had the bug.
What you will notice
For nearly everyone: nothing. Object access is unchanged, ListBucket via bucket/* is unchanged, and correctly written bucket policies are unchanged.
The one visible change: a request that tries to delete the bucket, or change its policy, replication, lifecycle, versioning, or object-lock configuration using credentials whose only matching grant is an object-only bucket/* pattern now returns AccessDenied. Bucket tagging, default encryption, and event notification are not affected. That is the boundary being enforced. If a deployment genuinely depends on the old behavior, set MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on and grant those actions on the bare bucket ARN (arn:aws:s3:::bucket) at your own pace.
What we deliberately left open
The general problem in #20449 — that bucket/* reaches the remaining bucket-level actions: ListBucket, GetBucketLocation, the configuration reads, CreateBucket, and the three tenant-plausible writes above — is not fixed here. Closing it fully means revoking grants that real deployments depend on, so it belongs to a future release that carries a migration path.
What that release owes operators is more than a wider action list, because no one can enumerate every deployment’s policies — which means shrinking or growing the protected set by guessing is an exercise with a hard ceiling. Three things raise it:
- A startup policy audit. Walk the stored policies and name each one whose meaning changes, in both the grant and the deny direction. That turns an upgrade surprise into a pre-upgrade checklist, it is read-only, and it can ship before the enforcement change rather than with it.
- A denial that explains itself. When a request is refused because only an object-scoped grant matched, say exactly that, and name the compatibility switch. A break an operator can diagnose in thirty seconds costs an order of magnitude less than a silent one.
- A switch with a scope.
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCHis all-or-nothing today: an operator who needs one action back has to reopen the self-escalation path along with it. Per-action scoping is what makes the change safe to adopt.
Recording the boundary rather than implying it: today twelve bucket-level writes are corrected. Everything else — the read/list family, CreateBucket, and bucket tagging, encryption, and notification — still honors bucket/* as a bucket-level grant, by decision, until that migration-gated change lands.
Closing
A single appended slash turned “only the objects” into “and the bucket too.” The tempting fix removes the slash everywhere and, in doing so, breaks a listing pattern half the world relies on and quietly weakens every Deny written against bucket/*. The fix we shipped removes it in exactly the place where an object-scoped grant should never have reached — the writes that can make a bucket public, and the ones that can delete it — and nowhere else. The rest is written down, waiting for a release where breaking it is something users are told to expect rather than something that happens to them.
Follow-up checked September 16: the “CORS has no handler” zero-cost argument describes early August. #71/#80 implemented per-bucket CORS, included in Server 20260903; Put/DeleteBucketCors remain protected bucket-configuration actions and must not move to the unprotected list. MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on is read during process initialization: supply it in the actual startup environment, not a later-loaded MINIO_CONFIG_ENV_FILE.
3.16 - Absent Is Not Empty: A Blank versionid and the Fail-Open It Invites
Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work.
Classification: Policy-enforcement correctness — a fail-closed report, a fail-open trap avoided, and one narrow trim bypass closed. Not a headline CVE — see How we classify this
Affected scope: Any deployment with a bucket/IAM policy using Null or StringEquals on s3:versionid; the reported break is on DeleteObject/DeleteObjects
Tracking: upstream minio/minio issue #21735 (reporter iTrooz, 2026-01-10); upstream repository archived read-only since 2026-04-25
This article is now public. The primary fix shipped as recorded above; residual findings below are the investigation-time record, not a claim that publishing one fix closed the entire defect class.
Conclusions first
- The policy engine decides
Nullby slice length, not by content. MinIO wrote"versionid": {""}into the condition map unconditionally, so a request that named no version still presented a length-1 slice.Null:{s3:versionid:true}— “match only when the key is absent” — could therefore never match, andNull:falsealways matched. The reporter’s “allow deletes only of the current object” policy denied every current-object delete (HTTP 200 envelope, per-objectAccessDenied). - The one-line fix is a trap. “Write the key only when it is non-empty” fixes the report and simultaneously opens something worse.
DeleteObjectscarries each object’s version in the XML body; the condition builder reads only the query string. Drop the empty key and a body version simply vanishes from the map — read as absent, i.e. as null — so a policy meant to protect old versions would authorize deleting a specific one. Fail-closed defect, meet fail-open bypass. - The real fix has two parts: write the key only when a version is named, and bind it, for
DeleteObject, to the effective server-resolved version (ReqInfo.VersionID) — the per-entry body value that the DeleteObjects loop already resolves — rather than to whatever the query string happened to carry. - A third, adjacent hole closed on the way: the builder read the version untrimmed while the object layer trims it, so a padded
?versionId=V%20let aDeny StringEquals s3:versionid "V"be sidestepped on the read/tag/copy paths. - Inherited from upstream, and unfixable there.
minio/miniois archived read-only, so the fix lives in the fork; this is the samegetConditionValueswe hardened in the condition-source work.
Absent is not empty
A condition key in a MinIO policy resolves to a lowercase name in a map[string][]string, and the engine answers Null by asking how long that slice is (silo-pkg .../policy/condition/nullfunc.go):
The content of the strings is never read. A slice {""} has length 1. To this function, a present-but-empty value is indistinguishable from a real version ID, and both are the opposite of absent.
Now the value that fed it, as inherited (cmd/bucket-policy.go, getConditionValues):
vid is the request’s ?versionId, empty on the overwhelming majority of calls. So every request, versioned or not, arrived at the engine carrying versionid: [""] — permanently length-1, permanently “present.”
The two Null directions then invert:
| Request | Map state | Null:true (want absent) |
Null:false (want present) |
|---|---|---|---|
| no version named | {""} (len 1) |
false — never matches | true — always matches |
?versionId=abc |
{"abc"} (len 1) |
false | true |
| (correct behaviour) no version | absent (len 0) | true | false |
The reporter wrote the canonical “let clients delete current objects but not roll back versions” policy — Allow s3:DeleteObject with Condition {"Null": {"s3:versionid": "true"}} — and watched every version-less delete return AccessDenied. The Allow never fired because its condition tested “no version named” and the map insisted a version was always named. StringEquals cannot see the difference either ({""} and absent both fail to intersect a non-empty policy value); only Null and ForAllValues:* are sensitive to it, which is why Null is where it surfaced.
The fail-open next door
The obvious fix writes the key only when it is non-empty, and for a single DeleteObject that is completely correct: no version → absent → Null:true matches. Ship that alone, though, and Multi-Delete turns it into an authorization bypass.
DeleteObjects (POST /{bucket}?delete) does not put versions in the query. Each object carries its own optional version in the request body:
The condition builder reads r.Form — the query string — and nothing merges an XML body into it. So under the naive fix, an entry that names version a1b2… in the body produces an empty query version, the key is omitted, and the engine sees absent — null. A policy written to allow only null-version deletes now matches, and the specific old version the operator meant to protect is deleted. The fail-closed nuisance from the report has become a fail-open on exactly the operation that most needs to be scoped per object.
This is the crux the reporter’s simple case hides: the condition value must be the version the server will actually act on for this object, and for Multi-Delete that value lives on a channel the condition builder never looked at.
The fix: the effective version, not a convenient one
Two mechanisms, because either alone is wrong.
1 — Represent absence honestly (cmd/bucket-policy.go). Write the key only when the request names a version, so “no version” becomes a length-0 read:
2 — Bind DeleteObject to the effective version (cmd/auth-handler.go, authorizeRequestWithTags). The DeleteObjects loop already resolves each entry’s body version into ReqInfo.VersionID (via checkRequestAuthTypeWithVID, cmd/bucket-handlers.go:502, a sequential loop — no shared-state race). Authorization rebinds the condition value to that server-resolved string, and deletes the key when it is empty:
An end-to-end test drives a &versionId=query-level-decoy on the DeleteObjects URL and asserts it never reaches any entry’s decision — the per-entry body value wins, the decoy is stripped.
Why DeleteObjectAction only, and not a blanket ReqInfo rebind. The tempting simplification — “always use ReqInfo.VersionID” — breaks copy. For a CopyObject, the source read is authorized as GetObject against the source’s version, which travels in the x-amz-copy-source header, and getConditionValues already extracts it there; ReqInfo.VersionID for a copy holds the destination query (usually empty). A blanket rebind would overwrite the correct copy-source version with the wrong one. Every non-delete version-aware operation (Get, Head, tagging, retention, copy-source read) carries its version in the query or the copy-source header, both of which the builder reads, and both of which are the effective version for a single object. Only Multi-Delete diverges. So the override is precisely as wide as the divergence, and no wider.
The version the server acts on is the trimmed one
One gap remained once the delete paths were correct. The builder read the version raw:
while every path that actually uses the version trims it first — newContext (cmd/utils.go:806) and getOpts (cmd/object-api-options.go:101) both strings.TrimSpace. So on non-delete version-aware actions, a padded ?versionId=V%20 presented "V " to the policy engine while the object layer read, tagged, or retained version "V". A Deny keyed on StringEquals s3:versionid "V" — “protect this exact version” — saw "V ", failed to match, and did not fire; the operation on "V" proceeded. A narrow bypass (the attacker must know the version and that a space changes nothing downstream), but a real one.
The fix trims both reads, aligning the condition value with the effective version:
DeleteObjectAction was already immune, because it uses the already-trimmed ReqInfo.VersionID. Trimming introduces no new allow: it can only make the condition value equal the version actually operated on, which tightens Deny and corrects Allow in the same direction. We proved it is load-bearing by removing only the trim and watching the padded test case go red.
What it affected
The reported break is on delete, but the underlying key is read by many actions. After the fix, every version-aware chain evaluates s3:versionid against the version the server resolves for that operation:
| Call chain | s3:versionid source |
Effective |
|---|---|---|
Single DeleteObject |
ReqInfo.VersionID = trimmed query, via override |
✓ |
DeleteObjects, per entry |
ReqInfo.VersionID = XML-body version, via override |
✓ — the fail-open closed |
GetObject / HeadObject / Select |
query, now trimmed | ✓ |
| Object tagging / retention / legal-hold | query, now trimmed | ✓ |
CopyObject / CopyObjectPart source read |
x-amz-copy-source version, now trimmed |
✓ |
| Anonymous 404-vs-403 probes | query (read-only) | ✓ |
| Admin / KMS / metrics / STS | no version concept | ✓ |
A forgery route was already closed by the earlier condition-source work and is worth restating: versionid is a reserved internal key (both the versionid and canonical Versionid spellings), so a client cannot inject a second copy through the header/query merge loops. The wire parameter is spelled versionId (capital I) and lands in an inert args["versionId"] the engine never reads.
Two directions of impact, kept distinct because they have different severities:
- Functional (the report): version-less deletes were wrongly denied. Fail-closed — an availability and usability defect, not a grant.
- Security (the trap and the trim): the naive fix would have granted deletes of protected versions on Multi-Delete (fail-open); and the untrimmed value permitted a narrow
Denybypass on read/tag/copy. The fix closes the first before it can exist and the second where it already did.
How we classify this
We are not minting a CVE for this, and the honest reasons are worth stating.
The behaviour the reporter filed is fail-closed: MinIO denied operations the policy meant to allow. A system that is too strict leaks nothing and grants nothing; it is a correctness and usability defect, and inflating a false-deny into a vulnerability would cheapen every real entry in this chronicle, whose neighbours are authentication bypasses and path traversals.
What carries genuine security weight is not the report but its vicinity. The fail-open on Multi-Delete is real, but it is a hazard we would have introduced, not one that shipped — the value of the two-part design is that the dangerous version never existed in a build. The trim bypass did exist, but it is narrow: it requires a Deny keyed on an exact s3:versionid, an attacker who knows the version, and it only ever affected non-delete paths. We closed it because it was in reach, not because it was a headline.
So: policy-enforcement correctness, filed here because that is where we keep silent enforcement failures, with the security interest recorded plainly rather than dressed up.
The boundaries we did not cross
Two same-class residuals remain, recorded rather than silently left:
- Multi-Delete governance-bypass follow-up. The adjacent gap was fixed by
75a6734e4/ #104 and shipped in Server 20260903, using each XML entry’s effective version for reauthorization. The Snowball observation below is a separate scope. - Snowball tar extraction.
PutObjectExtracttakes each member’s version from the tar PAX recordminio.versionIdafter the per-file authorization, so a named version can be written that never appeared in any condition value.
Both are narrow, both are pre-existing, and both would widen the change from “fix the reported key” into “re-plumb every action’s version into ReqInfo.” We scoped to the reported surface and wrote the IOUs down here, for the same reason the previous article recorded its object-layer omission: a deliberate omission that is not written down is indistinguishable from an oversight six months later.
A related decision, declined: the sibling keys username, userid, signatureversion, and authType are still written unconditionally empty, carrying exactly the present-but-empty defect versionid just shed — Null:{aws:username:true} is always false, including for the anonymous caller it should match. Fixing them is a one-liner each and a forty-caller blast radius, and some (principaltype is never empty) do not share the bug at all. We did not bundle a broad presence sweep into a versionid fix; it is named here as the next thread to pull.
Falsification
Three experiments, in the discipline that a test you have not watched fail is not yet a test.
- Revert both source files to
HEAD. The end-to-end DeleteObjects test turned red with every version-less entry returningAccessDenied— a faithful reproduction of issue #21735 — and the unit test caught the{""}key directly (“an absent versionId was exposed to policy evaluation”). Reapply, green. - Remove only the
TrimSpace. The padded case went red on the exact assertion —got [7f4b6b5f-…dd8 ]— proving the trim is not decoration. Restore, green. - The decoy. The Multi-Delete test appends
&versionId=query-level-decoyto the URL and asserts it reaches no entry’s decision, which is what distinguishes “reads the query” from “reads the effective per-entry version.”
The change touched only five files (cmd/bucket-policy.go, cmd/auth-handler.go, two tests, one doc example), committed with explicit paths in a working tree that had concurrent unrelated work in it, so nothing from the neighbouring efforts was swept in.
Source and lineage
The report is upstream minio/minio#21735, opened 2026-01-10 against RELEASE.2025-09-07T16-13-09Z: a Null:{s3:versionid:true} policy denying version-less DeleteObjects. The upstream repository went archived and read-only on 2026-04-25, so there is no upstream fix to wait for and no maintainer to coordinate with — the fork is the only venue, and the record here is the resolution.
The defect is old and inherited. getConditionValues has written versionid unconditionally for as long as the key has existed; the length-based Null semantics are upstream’s, in the policy package the fork consumes via silo-pkg. This is the same function and the same lineage as the earlier condition-source hardening that stopped client input from shadowing server-derived condition values — a related read of “what a policy condition is allowed to believe about a request,” continued here into “and it must believe the version the server will actually act on.”
Closing
Absent is not empty. A map that cannot say “no version” by leaving the key out will say it by leaving the value blank, and a Null that counts length will believe a version was named on every request that named none.
If one sentence survives: a fail-closed bug is the dangerous kind to fix, because the obvious repair flips it to fail-open — so bind the condition to the value the server actually acts on, from the same channel the operation reads, not the channel that was convenient; and when you stop at the reported surface, write down the versions you left on the wrong channel, rather than trusting the next person to find them.
3.17 - Three Headers, One Lie: Making the Client Source Address Mean Something
Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work.
Classification: Opt-in hardening plus a documentation defect, not a vulnerability and not a regression; no CVE assigned. The underlying weakness is inherited from upstream and its default behaviour is unchanged here
Affected scope: aws:SourceIp policy conditions, the audit log remotehost field, S3 event notification Host, and the client shown by mc admin trace — on any deployment whose S3 API port is reachable without passing through a header-sanitising proxy
Upstream: nothing to file — minio/minio is archived. Prior art there: PR #4736 (2017, the concern raised and half-addressed), discussion #17878 (2023, maintainer marks it working as intended), PR #20977 (2025, the partial switch)
This article states plainly that an
IpAddresspolicy condition is not enforceable on a directly-reachable MinIO deployment, and that this remains true by default after the change. That is a property of upstream MinIO as shipped, not a defect introduced by the fork, and it has never been documented anywhere. Publishing it is the point.
Conclusions first
- MinIO reads the client’s address out of three interchangeable headers —
X-Forwarded-For,X-Real-IP, RFC 7239Forwarded— and never from the TCP connection unless all three are absent. That address becomesaws:SourceIpand the audit log’s client field, so whoever controls it controls both IP-based access control and the attribution of every logged action. - The one switch that existed,
_MINIO_API_XFF_HEADER=off, suppresses one of those three. An attacker’s response is to sendX-Real-IPinstead. Our own code comment was recommending it as the mitigation. - “Put MinIO behind a reverse proxy” is not sufficient, for two independent reasons: on Kubernetes an Ingress and a ClusterIP Service routinely coexist so the proxy is not the only way in; and the stock nginx recipe appends to
X-Forwarded-For, leaving a client-supplied entry in the left-most position — which is exactly where MinIO reads. - The fix is a new opt-in setting,
MINIO_API_TRUSTED_PROXIES, generalising a trusted-proxy mechanism this fork already built for LDAP STS rate limiting. Set to a list, forwarded headers are believed only from those peers and chains are walked right-to-left. Set tonone, nothing is believed. - The most consequential decision was one we reversed. The first implementation widened
_MINIO_API_XFF_HEADER=offto suppress all three headers. That was the only part of the change that could alter an existing deployment’s behaviour, and it was backed out. The switch keeps its exact upstream semantics, and upstream’sTestXFFDisabledis retained unmodified as the proof. - Net compatibility impact: none for any deployment that does not opt in.
- Adversarial review found four defects in the first implementation, including one that made the new setting silently ineffective for every deployment configured through an environment file.
What the address is actually used for
The value comes from a single function, handlers.GetSourceIPFromHeaders. Tracing its consumers is what turns this from a logging curiosity into a security question:
| Consumer | Why it matters |
|---|---|
aws:SourceIp (cmd/bucket-policy.go) |
Decides IpAddress / NotIpAddress policy conditions |
Audit remotehost |
The record used to investigate every other incident |
Event notification Host |
Flows to downstream consumers as fact |
mc admin trace client |
Operator’s live view of who is doing what |
Two of these are security-relevant in different ways. A forged aws:SourceIp is a live access-control bypass: an IpAddress condition meant to confine a principal to an office CIDR is satisfied by asserting an address in that CIDR, and a NotIpAddress deny is evaded by asserting one outside it. A forged audit address is quieter and arguably worse — it corrupts the record retroactively, it applies even where no IP-based policy exists, and nobody notices until they need the logs.
Note also that the value is never validated as an IP address in the default path. Forwarded: for="_gazonk" is accepted and returned verbatim; upstream’s own test asserts it.
The setting that looked like a mitigation
internal/handlers/proxy.go gates exactly one header:
Setting _MINIO_API_XFF_HEADER=off costs an attacker one line: send X-Real-IP instead of X-Forwarded-For. Worse, disabling X-Forwarded-For moves the trust to a header the operator has not thought about, so the switch can leave a deployment in a state its owner has not modelled.
Where did it come from? Upstream PR minio/minio#20977, whose entire stated motivation is:
Customer request to disable all XFF header handling, ping me in Slack for more details.
No security rationale, no mention of X-Real-IP or Forwarded, no public discussion of why one header was gated and two were not. The narrowness is an oversight, not a considered scope. That mattered for the design, because it meant nobody had decided the other two should stay trusted — but as we will see, it did not end up justifying a change to the switch.
Upstream knew, in 2017
The most interesting thing found while writing this up is that none of it is news to upstream. The history is a small lesson in how a security decision decays.
August 2017. IpAddress / NotIpAddress condition support is added in PR #4736. During review the maintainer, @harshavardhana, raises exactly the concern this article is about: X-Forwarded-For is trivially spoofed, the left-most entry is the client’s own, and using it for a security decision would let a malicious client reach objects. The contributor accepts it and removes X-Forwarded-For support entirely, leaving only X-Real-IP, on the stated reasoning that a proxy sets it and a client cannot manipulate it. Merged five days later.
That reasoning is half right, and its unstated half is the whole problem: X-Real-IP is untamperable only if the proxy in front overwrites it. Nothing enforced that, and nothing told operators it was load-bearing.
Today. X-Forwarded-For is read first, ahead of X-Real-IP. The 2017 decision did not survive; it dissolved across later refactors of the condition-value plumbing rather than being reversed on purpose. There is no commit that says “we are re-admitting the spoofable header into policy decisions” — which is precisely how this class of decay happens.
August 2023. In discussion #17878 an operator reports that source IPs behind a load balancer are unreliable. The maintainer’s answer is unambiguous: without reliable source-IP visibility, IP-based restrictions are impractical, and the recommendation is to compartmentalise by tag or namespace instead. Marked working as intended.
So upstream’s own position — stated by a maintainer, in public — is do not rely on aws:SourceIp. That is a defensible engineering stance. What is missing is anywhere an operator would encounter it: it is not in the policy documentation, not in the condition-key reference, and not near the setting that appears to make it safe. An IpAddress condition is accepted without complaint and behaves as though it works.
That gap is the actual defect being fixed here, and it reframes the change. The allow-list is not overturning an upstream judgement; it is offering the mechanism that would make the 2017 concern answerable, to the deployments that want it. The documentation is doing the heavier lifting: writing down a contract that has been implicit since 2017 and contradicted by its own switch since 2025.
Why “put it behind a proxy” is not the answer
This is the standard advice, and it fails in two common, independent ways.
The proxy is not the only way in
On Kubernetes an Ingress and a ClusterIP Service routinely coexist. The Ingress sanitises headers; the Service does not, and any pod in the cluster can reach it. The security boundary is assumed to be the Ingress but is actually the pod network. The same shape appears in Pigsty deployments, where a load balancer fronts MinIO while the service ports remain reachable on the internal network.
The canonical nginx recipe preserves attacker input
The near-universal snippet is:
$proxy_add_x_forwarded_for expands to $http_x_forwarded_for, $remote_addr — it appends to whatever the client sent. A client sending X-Forwarded-For: 1.2.3.4 causes nginx to forward 1.2.3.4, <real client>. MinIO takes the left-most element, which is the attacker’s.
So a correctly-proxied, hardened, no-direct-access deployment is still forgeable, because left-most parsing and append-style proxies are mutually incompatible. Only proxy_set_header X-Forwarded-For $remote_addr; (overwrite) is safe under the default mode, and that is not what operators copy from the documentation.
This is the finding that rules out a documentation-only fix. Deployment discipline cannot close it; the chain has to be read from the other end, which requires code.
The design
Rather than inventing a mechanism, we generalised one this fork already has. MINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIES — added during the LDAP STS throttling work — already implements CIDR allow-list parsing, catch-all rejection, and a right-to-left chain walk, scoped to rate-limit bucketing. The parser moved to internal/config and both paths now share it.
One setting selects the mode:
| Mode | MINIO_API_TRUSTED_PROXIES |
Source address |
|---|---|---|
| Untrusted (default) | unset | unchanged from today |
| Trust nobody | none |
always the TCP peer |
| Allow-listed | addresses and CIDR blocks | forwarded headers, only from listed peers |
Under the allow-listed mode, X-Forwarded-For and Forwarded are read right-to-left, stepping over entries that name a listed proxy, and the first remaining address wins. Each proxy appends the peer it actually saw, so an entry the client injected sits to the left of the one its proxy wrote and the walk stops before reaching it. Appending proxies become safe.
GetSourceScheme is deliberately untouched. It feeds the Location URL in S3 responses rather than a policy decision, and suppressing it would hand http:// URLs to every deployment terminating TLS at a proxy.
Trade-offs
Whether to change the default
Changing the default to distrust forwarded headers would make every existing reverse-proxy deployment’s aws:SourceIp and audit addresses become the proxy’s address overnight. Policies could fail closed; audit continuity would break.
Not changing it leaves directly-reachable deployments exposed.
Decision: do not change it. But “do not change” is not the same as “stay silent”. The consequence is now written into the code comment and the operator documentation in as many words: under the default mode, an IpAddress condition is not access control and the audit address is not evidence. Making the cost visible so operators can choose is better than making a choice for them that detonates during a release window.
Whether to widen the existing switch
This is the decision we got wrong first and reversed, and it is the part of the story most worth recording.
The original brief asked that when an operator explicitly disables forwarded-header trust, clients must not be able to forge through an equivalent header. The obvious reading is “fix _MINIO_API_XFF_HEADER=off so it covers all three”, and that is what the first implementation did.
The case for widening was decent. Upstream’s PR title says “disable all X-Forwarded-For header handling”; its described scope is audit logs and IP-based access control, both of which are about not believing client-claimed addresses; the switch is undocumented, so its audience is small; and the failure direction is safe, since the fallback is the real TCP peer rather than an attacker-controlled value.
The case against turned out to be decisive. Widening it changes behaviour for a real population: operators whose proxy appends to X-Forwarded-For (polluted) but overwrites X-Real-IP (clean) may have discovered off as a way to get the correct address. Upstream’s TestXFFDisabled asserts exactly that behaviour — with the switch off and both headers present, X-Real-IP wins — so the behaviour is not merely incidental, it is pinned by a test. Those operators would have seen audit addresses silently flip from the real client to their proxy, and IpAddress conditions potentially start denying.
The reversal came from separating the goal from the mechanism. The goal was “a complete, enforceable way to turn this off exists”. Nothing required that the existing variable be the thing that provides it. Expressing it as MINIO_API_TRUSTED_PROXIES=none achieves the identical guarantee with zero effect on anyone who has not opted in.
Decision: leave _MINIO_API_XFF_HEADER exactly as upstream defined it. It gates X-Forwarded-For only, within whichever trust mode is in force. Upstream’s TestXFFDisabled is retained unmodified and still passes. The documentation now says plainly what the switch is not: it is a parsing switch, not a trust boundary, and a client refused one header simply sends another.
A secondary benefit: this collapses two interacting variables into one policy setting, so there is no longer a precedence rule ("off outranks the allow-list") for operators to learn and for us to get wrong.
Environment variable or config subsystem
Registering trusted_proxies as an api subsystem key would give mc admin config visibility, help text, and hot reload, matching how sts_trusted_proxies is done.
It would also introduce a window. Config subsystems load after the object layer initialises, so between process start and config application the trust policy would be empty — which under a “list is empty means trust everyone” reading is fail-open. A security boundary must not have a fail-open window. Separately, a trust boundary that can be changed at runtime is not obviously desirable.
Decision: environment variable. The cost is discoverability, and one bug described below.
Loopback is always trusted as a peer
The FTP and SFTP front-ends connect to the S3 layer over 127.0.0.1 and declare their session’s client with X-Forwarded-For (cmd/sftp-server-driver.go). An allow-list that does not exempt loopback attributes every FTP and SFTP request to the server itself.
The cost is that any process on the same host can forge. That is acceptable: an attacker who can open connections from localhost already has code execution on the host, and the threat model is lost well before this point. The FTP/SFTP regression, by contrast, would be certain and would affect everyone using those front-ends.
Decision: exempt loopback as a peer. Adversarial review then caught that the first implementation also treated loopback as a skippable chain entry, which is unnecessary for the FTP/SFTP case and actively harmful — see below. The two are now separate checks.
X-Forwarded-For versus X-Real-IP: genuinely unresolvable
Under the allow-listed mode, which header wins when both are present?
- A proxy that authors only
X-Real-IPand relays the client’sX-Forwarded-For(some nginx configurations) → preferringX-Forwarded-Fortakes the forged value. - A proxy that authors only
X-Forwarded-Forand relays the client’sX-Real-IP(AWS ALB) → preferringX-Real-IPtakes the forged value.
Both are common, and the server cannot tell which situation it is in from the request. This is not an undecided question; it is undecidable without the operator telling us what their proxy authors.
Decision: prefer X-Forwarded-For. It is the only one of the two that carries a chain that can be checked against the allow-list, and this path decides access control rather than rate-limit bucketing, so the value that can be validated should win. It also keeps header precedence identical to the default mode, so switching modes does not silently change precedence as well.
This deliberately diverges from getSTSLDAPTrustedProxySourceIP, which prefers X-Real-IP. Two contradicting implementations of the same question in one codebase is a hazard in itself, so both sites now carry a comment naming the divergence and its reason, so that nobody “unifies” them without re-deciding. The operator documentation states the mitigation for both directions: strip whichever header your proxy does not author.
A broad allow-list is worse than no allow-list
This is the most counter-intuitive property and the one most likely to bite.
Entries on the list are skipped during the chain walk. So MINIO_API_TRUSTED_PROXIES=10.0.0.0/8, configured because the load balancer is 10.0.0.1, makes every client inside 10/8 skippable as well. A client at 10.5.5.5 sending X-Forwarded-For: 8.8.8.8 produces a chain of 8.8.8.8, 10.5.5.5; the walk steps over 10.5.5.5 as a “trusted hop” and returns 8.8.8.8.
A broad list therefore does not merely trust more peers — it lets those peers forge. nginx’s set_real_ip_from has the same property with real_ip_recursive.
There is no algorithmic fix: the list is doing double duty as “who may forward” and “whose address may be discarded”, and separating them would mean two lists to keep in sync. Decision: keep one list, and make the constraint prominent — a callout block in the operator documentation, and the rule stated in the code comment where the walk happens. Only /0 is rejected as a catch-all, and the documentation says explicitly that this is a guardrail rather than a proof, since 0.0.0.0/1,128.0.0.0/1 covers the same ground.
Multi-node forwarding
MinIO forwards requests between nodes for bucket-DNS routing, listing continuation, heal-by-token, batch jobs and pool decommissioning. The receiving node’s TCP peer is the forwarding node, not the client.
| Mode | Receiving node resolves |
|---|---|
| Default | the client |
none |
the forwarding node |
| Allow-list without node addresses | the forwarding node |
| Allow-list with node addresses | the client |
This is not an obscure path: a ListObjectsV2 continuation token carries the node index, so any client can cause its own request to be forwarded. Under none, that request is then evaluated with aws:SourceIp set to an internal node address — which an IpAddress condition allowing internal ranges would treat as a pass.
Decision: document it, and steer multi-node clusters to the allow-list. none cannot be corrected for this case, because it believes nothing by definition. Automatically seeding the cluster’s own addresses was considered and rejected: it needs DNS resolution at startup and re-resolution as node addresses change, which is more machinery and more failure modes than the explicit configuration it replaces.
Compatibility
The change was deliberately structured so that risk is not spread evenly across it. Every piece is either opt-in or dead code in the default configuration.
| Change | Who is affected | Risk |
|---|---|---|
MINIO_API_TRUSTED_PROXIES allow-list |
only those who set it | none |
Forwarder sanitises X-Real-IP / Forwarded |
code path does not execute in default mode | none |
| Startup failure on a malformed value | only those who set it, incorrectly | none |
| Policy re-read after environment-file load | same result when nothing is set | none |
| LDAP parser extracted for sharing | nobody — pure code motion, verified identical | none |
_MINIO_API_XFF_HEADER semantics |
nobody — reverted | none |
_MINIO_API_XFF_HEADER read timing |
nobody — upstream timing kept deliberately | none |
Evidence for the default path. unverifiedSourceIP is a verbatim copy of the original function body, including its quirks: the ", " separator, the fall-through when the left-most element is empty, and the acceptance of non-IP values such as _gazonk. An independent review verified behavioural parity against HEAD over 21 cases — empty X-Forwarded-For, a bare comma, a leading ", ", "," versus ", " separators, " , ", IPv4-mapped addresses, bracketed IPv6, non-IP junk, and all three Forwarded forms. Upstream’s TestGetSourceIP and TestXFFDisabled are both retained unmodified and pass.
A subtlety self-review caught. The new setting is read after MINIO_CONFIG_ENV_FILE is loaded, which is what makes it work in packaged deployments. The obvious tidiness move is to read _MINIO_API_XFF_HEADER in the same place — and that would have been a behaviour change, because upstream reads it at package initialisation, before environment files exist. An operator who wrote it into an environment file has it silently ignored today; picking it up would make an already-deployed setting suddenly start working, flipping their source addresses from the left-most X-Forwarded-For entry to X-Real-IP. The old switch therefore keeps upstream’s read timing along with upstream’s semantics, and a test pins that so nobody tidies it later. The quirk is documented instead: set it in the process environment if you want it honoured.
The defensive code that was not defending anything. Sharing the parser initially came with two extras: allow-list entries written in IPv4-mapped form were rewritten to the IPv4 prefix they denote, and the address being matched was unmapped and de-zoned. Both looked like corrections — an entry written ::ffff:192.168.1.10 is otherwise accepted and then matches nothing, which is a silent failure worth removing.
They were removed anyway, and the reason is worth recording. Both call paths reduce the address through net.ParseIP(...).String() before matching, and that already collapses ::ffff:10.0.0.1 to 10.0.0.1; a dual-stack listener reports an IPv4 peer in plain form regardless. So neither extra could be reached by a real request. Their only observable effect was on what the shared function meant for the LDAP STS allow-list that had been using it first — 18 differences that a test could see by calling the function directly and no deployment could.
Worse, one of them manufactured the fail-open described below: rewriting ::ffff:0:0/96 turned a /96 into 0.0.0.0/0. Deleting the rewrite removes the bug’s cause rather than ordering around it. What remains is pure code motion, verified identical to the previous implementation across every combination of 37 allow-list values and 21 peer addresses — zero parse differences, zero match differences. The wart it declined to fix (a mapped-form entry matches nothing) is the pre-existing behaviour, fails closed, and is now stated in the function’s own comment so the next person does not re-derive the same tempting fix.
The one residual risk worth naming. The default mode’s code path did change: there is now a switch and a function call in front of the original body. If that plumbing were wrong it would affect everyone, not just opt-in users. The parity testing above is why we believe it is not, but “verified equivalent over 21 cases” is a different claim from “provably identical”, and the honest version is the former.
What adversarial review found
An independent agent was tasked with breaking the first implementation. It found four real defects, all since fixed and covered by regression tests.
A silent fail-open, and the worst of the four. The trust policy was read in the package’s init(). But loadEnvVarsFromFiles() calls os.Setenv for everything in MINIO_CONFIG_ENV_FILE long afterwards — which is how MinIO is configured in essentially every packaged deployment. An operator putting MINIO_API_TRUSTED_PROXIES in /etc/default/minio would have got the historical trust-any-peer mode, with no error reported, and a malformed value would have been silently ignored rather than fatal. The policy is now applied in serverHandleEnvVars, which runs after the file load and before any listener.
Loopback skipped as a chain entry. Described above: the peer exemption was being reused as a hop exemption, which is a needless instance of the broad-list problem. Now two separate checks.
Two fail-closed correctness bugs. A zoned IPv6 peer (fe80::1%eth0) canonicalised to nothing, because net.ParseIP rejects zones — so such a peer could never be a trusted proxy. And an allow-list entry written in IPv4-mapped form (::ffff:192.168.1.10) was accepted at startup and then matched nothing at all, since netip.Prefix.Contains is false across differing bit widths.
Two further defects were found while writing the documentation rather than the code, which is its own small lesson:
Repeated header lines. Header.Get returns only the first header line. HAProxy’s option forwardfor adds a second X-Forwarded-For line rather than extending the first, so Get would hand back the client’s line and put the forged value right back where the right-to-left walk exists to avoid it. Now flattened across all lines with Header.Values.
The internal forwarder relayed client claims. internal/handlers/forwarder.go set X-Real-IP only when absent, so a client’s value passed between nodes unchanged. Under an allow-list that includes the cluster’s own nodes — the configuration we recommend — a client could thereby borrow a peer node’s authority. The forwarder now drops X-Real-IP and Forwarded when the incoming peer is not entitled to have set them. X-Forwarded-For needs no such handling, because Go’s ReverseProxy appends the true peer and the receiving node’s walk reaches that entry first.
A second round, after the rework
Reworking the setting warranted a second adversarial pass, which was worth running: differential testing found zero behavioural differences against HEAD across 4,745,520 source-IP resolutions and 345,600 forwarder rewrites, but it also found three more ways to fail open — one of them introduced by the first round’s own fix.
A catch-all smuggled in as an IPv4-mapped prefix. MINIO_API_TRUSTED_PROXIES=::ffff:0:0/96 is a /96 as written, so it passed the catch-all check; the rewrite that unmapped IPv4-mapped entries then turned it into 0.0.0.0/0, trusting every peer. The first fix moved the breadth check to the far side of the rewrite. The eventual fix deleted the rewrite, once it became clear it was unreachable by any real request — which removes the cause instead of guarding its output.
This is the one most worth dwelling on. The fail-open was manufactured by a fix for an unrelated fail-closed bug, and the fix for the fail-open was a reordering that left the manufacturing step in place. Two rounds of correction, both defensible, neither addressing the fact that the code should not have been there. Hardening changes deserve the same adversarial treatment as the code they harden, and “is this reachable at all?” belongs near the front of that treatment.
A deliberate value naming nobody. MINIO_API_TRUSTED_PROXIES="," parsed to an empty list and fell back to the permissive default. An empty unset variable must mean “default”, but a value the operator actually typed which names no proxy is a mistake, and answering it with trust-everyone is the one behaviour they cannot have wanted. It is now a startup error. Whitespace-only remains equivalent to unset, since that is what an empty shell variable expands to.
A remote value that could not be read. MinIO supports env:// indirection, where a variable’s value is fetched from a remote webhook. env.Get discards the error from that fetch and returns the empty string — which this code would have read as “unset”, reinstating trust-any-peer at exactly the moment the operator’s intent could not be determined. The setting is now read through env.LookupEnv so the error is surfaced and startup stops. This is a general hazard for any security-relevant setting read via env.Get, and worth remembering beyond this change.
A third pass, attacking from angles the first two shared
Both earlier passes attacked the resolver as a unit. Two things that neither could see:
Nothing had tested that the trust policy reaches a decision. Every test to that point checked what the resolver returned, and the one test at the policy layer only asserted that aws:SourceIp equalled the resolver’s output — in the default mode. So a version where the resolver was correct but the policy engine read something else would have passed everything. There is now a test that drives a forged X-Forwarded-For through getConditionValues into a real IpAddress evaluation under each mode: believed by default, ignored under none, ignored from an unlisted peer, and still honoured from the listed proxy. It passes, but it should have existed before the change was called done.
The allow-listed mode had a resource amplification the default mode does not. The chain was flattened into a slice before being walked, so a client behind a trusted proxy could turn the 1 MiB header allowance into roughly 33 MB of slice headers per request — around thirty-fold — plus a million-iteration walk. The default path never had this, because it uses strings.Index on the raw header. The walk now scans backwards over the header text in place, allocating nothing, and stops after 100 hops; real chains are a handful, the answer sits at the right-hand end, and running out of budget yields no address, which falls back to the peer. A test pins the zero-allocation property, because it is the kind of thing an innocent-looking refactor would undo.
Also corrected in this pass: the deployment contract was stated too narrowly. “The proxy must overwrite whichever headers it sets” misses the case that actually bites — a proxy that correctly authors only X-Real-IP or only Forwarded still relays the client’s X-Forwarded-For, and that is the header read first. The general rule, now stated as such, is to strip every source-address header the proxy does not itself write.
One reported finding was reviewed and not treated as a defect: the catch-all guard rejects /0 and nothing else, so 0.0.0.0/1,128.0.0.0/1 covers the same ground and is accepted. Tightening it would mean rejecting broad-but-not-/0 prefixes in the parser now shared with the LDAP allow-list, newly failing configurations that are valid today, to defend against a value no deployment realistically holds. The documentation states plainly that the check is a guardrail rather than a proof, and the callout about naming proxies instead of subnets is where the real defence lives.
Recommendations
By topology:
- Proxy you control, API port genuinely unreachable otherwise. Set nothing. But verify whether your proxy overwrites or appends: if the config says
$proxy_add_x_forwarded_for, you are forgeable today. Switch to$remote_addr, or adopt the allow-list. - Direct exposure, no proxy.
MINIO_API_TRUSTED_PROXIES=none. - Kubernetes or Pigsty, Ingress plus reachable Service. The allow-list, containing the proxy addresses and the MinIO node addresses. This is the configuration that makes an
IpAddresscondition mean anything. - Any multi-node cluster. The allow-list with node addresses, not
none.
Two rules apply to every allow-list deployment. Name proxies, not the subnet they occupy. And strip at the edge every source-address header your proxy does not itself write — listing a peer means believing all three headers from it, they are consulted in a fixed order, and a header your proxy leaves alone is entirely the client’s. A proxy that correctly authors only X-Real-IP, or only Forwarded, still relays the client’s X-Forwarded-For, which is read first.
What was not done
- Automatic seeding of cluster node addresses. Feasible via
EndpointServerPools, but it needs DNS resolution and re-resolution on address changes. Explicit configuration was judged the smaller risk. - Registration as an
apiconfig key. See the fail-open window above. Revisitable if observability turns out to matter more than the startup guarantee. ExistingObjectTag/*. This adjacent condition-source defect was subsequently fixed by2f55347f7and shipped in 20260804: only loaded stored tags supply this condition, not client request tags. This does not add tag loading to every S3 operation.
Verdict on severity
Default behaviour matches upstream, and MinIO has never documented aws:SourceIp as trustworthy on a directly-reachable deployment, so “the default is unsafe” is closer to a documentation defect than a vulnerability. One thing is squarely a defect, though: an operator set an explicit security switch and it did not do what its name and its only public description implied, and it failed silently. That is worth a record, scoped to the incompleteness of upstream’s _MINIO_API_XFF_HEADER rather than to anything the fork introduced.
minio/minio is archived, so there is no upstream to coordinate with — the same position as CVE-2026-42600.
3.18 - Sorted Is Not Increasing: How One Duplicate Part Number Doubled an Object
Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work. Classification: Data correctness, not a vulnerability — see Why this is not a CVE Affected scope: All backends, any authenticated S3 client, on its own upload Tracking: pgsty/silo #49
This article is now public. The primary fix shipped as recorded above; residual findings below are the investigation-time record, not a claim that publishing one fix closed the entire defect class.
Conclusions first
sort.SliceIsSortedwith a<predicate does not test strict increase. It tests the absence of an inversion. Equal neighbours contain no inversion, so[1,1]was accepted.- Upload one 5 MiB part, complete with
[1,1], and the server returns HTTP 200 and a 10 MiB object. The upload is then consumed: a corrected retry getsNoSuchUpload. The client cannot recover. - Inherited from upstream, and old. The check has had this shape since 2016-08. Two refactors — 2017 and 2023 — rewrote it faithfully, because each preserved the predicate, and the predicate was never the problem.
- The fix is one loop at the handler layer. The object layer is left undefended by decision, and that IOU is written down here rather than left implicit.
- Three independent reviews found no defect in the fix. What they found was a comment that misstated why a neighbouring guard exists — and, through that comment, an unrelated node-level panic.
The verb, not the predicate
The code, as inherited:
It reads as “reject unless the part numbers strictly increase.” It does not do that. IsSorted evaluates the predicate in the reversed direction only — for each neighbouring pair it asks less(i, i-1), i.e. “is this element smaller than the one before it,” and reports unsorted the moment one such inversion appears. For a pair of equal elements that question is false. No inversion, therefore sorted.
The consequence is worth stating precisely, because it is what makes the misuse survive review: no strict predicate can make IsSorted reject duplicates. The only spelling that works is the non-strict one — passing <= as the less function, so that equal neighbours register as an inversion. To ask for strictly increasing you would have to write the operator that reads as not strict. Every reviewer who checked that the predicate said < was checking the right character in the wrong function.
Our replacement drops IsSorted rather than trying to spell it correctly:
The rejection-set delta is exactly one class: lists containing an adjacent equal pair. Everything previously rejected is still rejected; everything previously accepted, except duplicates, is still accepted. Non-adjacent duplicates come along for free — a strictly increasing sequence is globally distinct, so [1,2,1] and [1,3,2,3] are caught by the inversion they are forced to contain.
Two refactors preserved it faithfully
The archaeology is the most transferable part of this incident.
| When | Shape | What changed |
|---|---|---|
| 2016-08 | sort.IsSorted(CompletedParts(parts)) |
already present at server: Move all the top level files into cmd folder (#2490) |
| 2017-11 | same call, Less moved onto an exported type |
Add public data-types for easier external loading (#5170) |
| 2023-04 | sort.SliceIsSorted(parts, func(i,j) bool { … < … }) |
simplify sort.Sort by using sort.Slice (#17066) |
Both refactors were correct as refactors: they preserved behaviour exactly, which is what a refactor is supposed to do. The 2023 commit was a repository-wide cleanup with no bearing on multipart semantics at all. It carried the < across unchanged, and the < was never wrong — CompletedParts.Less needs < to be a valid sort.Interface.
The defect lived in the relationship between the predicate and the function it was handed to, and a refactor that moves the predicate cannot see that relationship. A decade, three shapes, one behaviour: an ordering check that answers a question adjacent to the one it appears to answer.
What it actually did
Measured against both erasure backends, through the real signed HTTP handler:
| Uploaded | Completion list | Response | Resulting object | ETag suffix |
|---|---|---|---|---|
| one 5 MiB part | [1,1] |
200 OK | 10,485,760 bytes | -2 |
| two 5 MiB parts | [1,2,2] |
200 OK | 15,728,640 bytes | -3 |
| one 5 MiB part at 10000 | [10000,10000] |
200 OK | 10,485,760 bytes | -2 |
The ETag suffix is the part count the server believes it assembled. There is no internal disagreement to detect: the metadata, the size, and the ETag are all mutually consistent and all wrong. The object simply is not what was uploaded.
Two properties make this worse than a bad error code.
The upload is consumed. Assembly runs to completion and cleans up the multipart upload, so the corrected retry returns NoSuchUpload. A client that notices the wrong size cannot fix it by resending the right list; it has to start the whole upload over, if it still has the data.
It is reachable by accident. No adversary is required. Any client that appends a part to its completion list twice — a plausible bug in a resumable-upload wrapper, a retry path, or a list built by concatenation — silently gets a doubled part instead of a 400.
Why this is not a CVE
It belongs in this chronicle because it is a silent server-side correctness failure, and this is where we keep those. It is not a vulnerability, and we are not going to inflate it into one.
The request must carry the caller’s own credentials, address the caller’s own upload, and the damaged object is the caller’s own. There is no cross-tenant effect, no privilege change, no disclosure, and no path to another account’s data. What breaks is the guarantee that a completed multipart object equals the bytes you uploaded — serious, but a correctness guarantee, not an access-control boundary.
The entries around this one in this chronicle are authentication bypasses and path traversals. Filing this beside them under the same label would make every label in the table mean less.
The boundary decision, and what it costs
The object layer has no duplicate defence at all. erasureObjects.CompleteMultipartUpload sizes its output slice to the request (cmd/erasure-multipart.go:1249) and then resolves each requested part number against current metadata (:1255). The same number resolves twice, writes two identical ObjectPartInfo entries, and adds its size twice. AddObjectPart does deduplicate by part number, but it deduplicates the metadata slice, not the request. The 5 MiB minimum-size rule cannot help either, because the duplicated part is individually legal.
We fixed the handler and left that alone. The reasoning:
- It is the only entrance where a client-controlled list exists. The other four callers — batch, restore, decommission, rebalance — build their lists server-side from
oi.Partsor1..n, and are strictly increasing by construction. - The required output is an S3 error code, which is an API-layer concern. The object layer’s error vocabulary maps to a different code, so intercepting lower would hand clients a less accurate diagnosis.
- Minimality. This fork ships narrow fixes, and a change in the assembly loop is not narrow.
The cost, recorded rather than implied: the uniqueness invariant now has exactly one enforcement point, and nothing enforces the enforcement. No compiler error and no test failure will greet the person who adds a fifth caller to the object layer; they will get a silently corrupted object. That is the same species of IOU the previous article recorded about getVolDir, and it is written down for the same reason: an unrecorded deliberate omission is indistinguishable from an oversight six months later.
What we deliberately did not add
Part numbers need not start at 1 and need not be consecutive. [1,3], [5,9] and [3] are all legal S3, and all still complete successfully.
This matters more than it sounds. “Also require the list to start at part 1” is a one-line addition that looks like tightening, would pass a casual review, and would break legal clients — anything that abandons a part after a failed upload and completes with what it has. The temptation is real precisely because the fix next door is about validating the same list.
So two test cases exist for no purpose other than to make that change fail. We verified they do their job by injecting the constraint and confirming that exactly those two cases went red and nothing else did. A guard rail nobody has fired once is a guess.
The one behaviour change we did not intend
A 14-input differential against the pre-fix build turned up exactly one behavioural change beyond duplicate rejection: [0,0] and [-1,-1] — lists that are both duplicated and out of range — moved from InvalidPart to InvalidPartOrder. Both are HTTP 400.
We accepted it, on the principle that a format error should outrank a state error: an ordering violation is decidable without reading any storage, while part existence is not. It also only affects requests that were going to fail regardless, so no client that previously succeeded can now fail.
On S3 fidelity itself we are making a documented inference, not a measurement. AWS defines InvalidPartOrder as the parts list not being in ascending order, and documents that part numbers may be non-consecutive; duplicates are not ascending. We did not verify this against a live AWS endpoint, and two independent reviewers reached the same conclusion by the same documentary route, which is agreement, not evidence.
Falsification, and a comment that was wrong
Two mutation experiments, in the discipline the previous article argued for — a test you have never watched fail is not yet a test.
Inject “must start at part 1.” Exactly the two gap cases went red; the four lists starting at 1 stayed green. The guard rail is targeted, not incidental.
Delete the neighbouring len(Parts) == 0 guard. The expected result was that an empty completion would produce some wrong-but-orderly error. The actual result was that the process panicked: the empty list reaches a storage decorator that indexes element zero of the part-path slice without a length check, on a goroutine that no recover can reach. The S3 face is masked by that one guard line, which has been there since 2022 and is not documented as load-bearing. It is tracked separately as an unfixed node-level defect, which is why this article is held.
And the part worth publishing at our own expense: the comment we wrote about that guard was wrong. It said dropping the length check would let an empty completion succeed — the opposite direction of the truth, and specifically the direction that understates danger. It was caught in review and corrected before the commit. A comment that misstates why a check exists is exactly how the check gets deleted three years later by someone tidying up.
Three acceptances, zero blocking findings
The change went through three independent gates before commit:
| Gate | Method | Outcome |
|---|---|---|
| Author | revert the fix, watch the test go red at the measured 10 MiB, reapply, watch it go green | red/green established |
| Independent reviewer | rebuilt the red state in its own detached worktree rather than trusting the report; 14-input differential | no blocking finding |
| External model, different vendor | read-only sandbox, independent derivation of the rejection-set argument and of the AWS reading | conditional accept; the condition was that it could not compile in its own sandbox |
Stated plainly, because the honest version is less flattering than the table: none of the three found a defect in the fix. What review produced was the corrected comment and, through the mutation it prompted, the discovery of the unrelated panic. That is still a good return, but it is not the same as catching a bug in the patch, and the record should say which one happened.
The rebuilt-red-state detail is the one worth copying. A reviewer who reruns the author’s tests is checking the author’s arithmetic; a reviewer who reconstructs the broken state independently is checking the author’s claim.
Declined, and left open
Declined, deliberately:
- Two test additions — completing onto a pre-existing object, and giving each part distinct content so ordering is verified rather than just total size. Both are real improvements. Both were declined under a standing rule that this fork ships correctness and security fixes rather than test expansion, and the core invariant is already pinned by “the rejected request left no object and the upload still works.”
- XML root element name is not validated. A document with the wrong root but correct
<Part>children is accepted. This is not a bypass — the same list still goes through the same check — it is pre-existing, and tightening it risks breaking real SDKs over namespace handling. Recorded, not fixed.
Left open, none of it in a released build as of 2026-08-03:
- Object-layer defence in depth for part uniqueness (see above).
- The empty-list panic in the storage decorator, tracked as a node-level defect.
- XML strictness, including
<PartNumber>abc</PartNumber>returning 500 where 400MalformedXMLis correct.
The concurrent checksum work on completion (#46, #48, #50) was fenced off from this change entirely and shares no code with it.
Closing
The predicate was strict. The verb was not. A decade of review read the predicate — including the two commits that rewrote the line.
If only one sentence survives: check what the function does with the comparison, not just what the comparison says, and when you decide to leave the layer underneath undefended, write it down where the next person will trip over it, rather than trusting that they will re-derive your reasoning.
4 - Design Records
Design records capture the reasoning behind SILO maintenance decisions: the problem being solved, the compatibility boundary, rejected alternatives, implementation requirements, and the evidence required before release.
Read the date and status first: a proposal describes work awaiting implementation or evidence; merged on main establishes source inclusion; released links an actual tag or release record. Historical decisions and deferrals do not automatically describe current behavior. Prefer fixed commits, PRs, stable tests and release records; distinguish local checks, remote CI, artifacts and production deployment.
4.1 - Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence
#77 is a reproduced site-replication correctness defect. A receiver replaces source time with arrival time and may then reject a genuinely newer deletion. Some configuration types stop exporting their timestamp after deletion, preventing heal from recovering a delete missed during an outage. Adding a DELETE branch alone cannot solve both problems.
Merge status (2026-09-12): the repair and research archive were merged into
mainthrough PR #180, commit 48ec10312. #77 is closed. All nine pre-merge CI checks passed.
Review boundary: the plan went through four Claude Code Opus 5 Max review rounds, passing the final two. Full implementation review, remediation review, and focused final acceptance returnedGO_WITH_NONBLOCKING_NOTESin all three rounds. Final blockers are zero; the requested full cmd and final lint checks have now passed.
Applicability: this page describes the repair now included in main. Check the specific version of downloads and running installations; full deletion recovery still requires every node to be upgraded and deletion export to be enabled consistently.
Existing work and scope
The earlier release notes, security hardening record, and Server compatibility page already document the #77 deletion limitation. They do not provide a complete record of its state model, alternatives, or verification boundaries. This page supplies that reasoning.
The source repository archive retains the original reproduction, plan versions, final reports and invocation identities for all seven review rounds, finding dispositions, executed test logs, source/binary hashes, and a rerunnable two-site driver. Raw model reasoning streams, binaries, and temporary lab volumes are excluded. Original artifacts and copies with normalized workstation paths and document links have separate hashes.
| Existing work | What it repaired | What it does not establish |
|---|---|---|
| #91 | Per-site configuration counts, Policy/Quota reporting, malformed-field isolation | Accurate counts do not prove convergence of values and source times |
| #103 | Serialized writes to the whole .metadata.bin record |
A lock cannot validate an ordering decision made before acquiring it |
| #76, #78 | Object Lock’s replication payload and existing-bucket adoption protection | They do not replace a consistent source-state comparison for six types |
| CORS replication repair | A separate per-bucket CORS deletion register and replication trust boundary | Its semantics cannot be applied blindly to every metadata type |
This repair covers Policy, Tags, SSE, Quota, Versioning, and Object Lock. Lifecycle/expiry has its own merged-payload time semantics; CORS keeps its separate mechanism. Notification, IAM, object replication, MRF, resync, and public counters are not rewritten. Object replication reliability has a separate record.
The supported release target is the maintained silo, silo-console, mc, and silo-pkg stack. Compatibility with unmodified upstream MinIO/MC remains best effort and does not require downgrading maintained components or recreating dependency forks.
What the reproduction established
The original regression reproduced on both ErasureSD and 16-disk Erasure ObjectLayers. A peer PUT originated at a past time T, but the receiving disk stored current arrival time. A subsequent DELETE carrying T+1 minute was rejected as older. RPC success alone therefore cannot establish correct final state.
| Configuration | Original PUT preserves source time | Established empty-event behavior | Original timestamp export without payload |
|---|---|---|---|
| Policy | No | Delete | Yes |
| Tags | No | Delete | No |
| SSE | No | Delete | No |
| Quota | No | Delete; zero-value JSON has separate semantics | No |
| Versioning | No | No operation | Not used as deletion |
| Object Lock | No | No operation | Not used as deletion |
Three further entry-point defects matter: an old bulk event can overwrite a newer field; a request that checks state before queuing for the lock can act on an obsolete decision; and remote Tag heal omits UpdatedAt. Each requires a repair at the actual entry point, beyond changing source selection in heal.
The facts a field needs
Reuse the existing payload, field UpdatedAt, and bucket Created. No disk format, SDK field, or persisted deployment ID is added.
| State | Conditions | Eligible source |
|---|---|---|
| Unknown or invalid | Unknown creation time, invalid payload, or field time before creation | No; missing information cannot mean deletion |
| Empty baseline | Empty payload at zero time or Created | No |
| Live baseline | Valid nonempty payload at zero time or Created | Yes, for historical configuration initialization |
| Real update | Valid nonempty payload later than Created | Yes |
| Real deletion | Empty payload for a deletable field, later than Created | Yes; this timestamped empty value is a tombstone |
Empty Versioning and Object Lock events remain no-ops. Sharing a helper must not give these types deletion semantics. Zero field times use Created as a comparison baseline; this does not turn historical emptiness into a new delete.
Ordering first gives real states priority over baselines, then compares source time between real states. At the same time, a deletion wins over a live value. Conflicting live values at the same rank use a stable content key, with the lexicographically greater key winning. For timestamped peer events and heal, an identical effective state causes neither a save nor a notification. A local write allocates a new monotonic revision even when the visible payload is unchanged. Live baselines also use content-key ordering; a later creation default cannot outrank a real change.
This is deterministic conflict resolution. It does not mean a lexicographically greater configuration better expresses business intent. Operators must still choose and resubmit the intended value after conflicting concurrent changes.
Comparison must match persistence
Policy sets are map-backed, so ordinary JSON encoding can depend on iteration order. The Server sorts the complete JSON tree of an already validated policy, including Statement, Action/NotAction, Resource/NotResource, Principal, and Condition. Sid and numeric precision are retained. This adds no syntax unsupported by the existing parser.
That parser accepts NotAction and NotResource, but the original structure’s required Action/Resource encoding can fail on the corresponding empty sets. Explicit field encoding addresses this, and Policy GET/admin export use it too. The purpose is to keep an accepted policy readable, beyond making its comparison key deterministic. GET/export now sort Statement arrays that formerly followed stored order, and sort set arrays whose map-backed order could vary between calls. Authorization is unchanged; upgrading does not rewrite every stored policy.
Quota keys use the existing parsed representation encoded as JSON. {}, JSON null, and valid zero-quota documents remain live documents, not implicit deletion events. Older senders converted a zero-quota PUT to deletion for peers while keeping a local document. mcli quota clear sends a zero quota; upgraded peers retain that live zero document, with no capacity enforcement either way. An empty Policy instead follows the established peer deletion interpretation: a valid empty-policy PUT succeeds, and a subsequent GET returns the existing NotFound response.
XML configurations use the bytes of a valid document. There is no general XML canonicalization layer. Versioning needs one exception: apply the existing Object Lock constraint before comparing the effective document that will actually be persisted. Otherwise comparison can accept a value that Save rewrites, causing heal to send it again next time.
Lock the decision as well as the save
The six fields share one .metadata.bin record. Loading raw state, validation, comparison, modification, and saving must all happen under the existing metadata.lock. Comparing outside the lock still permits stale decisions; separate field locks would allow whole-record read/modify/write operations to overwrite one another.
A local write allocates its time inside the lock:
This lets a local correction advance beyond an already stored future field time. Timestamped peer events retain their original time. Identical or older timestamped peer states return without a write; this is not a no-op guarantee for local PUT/Delete.
Bulk processing handles only explicitly supplied fields, validates them, and saves at most once. Omission preserves a field; explicit null follows its type’s semantics. An invalid field cannot leave half the bulk update persisted. Import assigns a common time for the selected six-type fields under the final per-bucket commit lock. Disk state and outbound events use that commit’s final snapshot, not a later read combined with an earlier time. A normalized empty Policy uses the existing dedicated deletion event so bulk omitempty cannot lose it.
The save helper returns its normalized snapshot. Public Update/Delete signatures stay unchanged, while other metadata retains its existing processing and notification behavior.
Adoption must not manufacture a deletion
Adoption can change Created. Moving it earlier while retaining an empty field’s old default timestamp makes an empty baseline appear to be a real deletion. Moving it later can invalidate historical initial values.
The narrow repair, under the existing adoption lock, rebases only these six fields whose previous time was zero or equal to old Created. Actual update and deletion times remain unchanged. An empty payload alone is not proof of a default, and existing-bucket configuration protection is not redesigned. Genuinely different bucket generations still require operator intervention.
One rule across the real entry points
| Entry point | Required behavior |
|---|---|
| Local S3/Admin writes | Monotonic time under the lock; use the committed snapshot for outbound events where needed |
| Typed peer events | Compare and persist original source time under one lock; retain existing types and legacy Object Lock payload compatibility |
| Bulk/import | Explicit field presence and atomic save; allocate import time at final commit |
| Initial synchronization | Preserve historical live baselines; include real deletions after opt-in |
| Local/remote heal | Use the same comparison for selection and apply, with complete source times; unknown IDs and failed peers do not block healthy targets |
| Status export | Value and time belong to one record; gate newly exposed deletion times during rollout |
Initial synchronization retains its existing five-type send path. Versioning is still initialized through MakeBucketHook and aligned through heal. No extra initialization path is added merely to make the table symmetric.
Heal filters valid candidates before selecting the maximum, instead of seeding from the first map entry and only then filtering defaults. Public mismatch counters cannot be the sole gate: equal content with different source times still needs synchronization. Conversely, equal effective states need no further write or RPC.
Why rollout needs a default-off switch
The startup setting MINIO_SITE_REPLICATION_METADATA_TOMBSTONES defaults to off. It controls visibility of newly exposed deletion information and does not detect remote capability.
| Behavior | off | on |
|---|---|---|
| Source-time ordering and atomic apply | Active | Active |
| Ordinary deletion events | Still replicated | Still replicated |
| Existing Policy deletion-time export | Preserved | Preserved |
| Real deletion-time export for absent Tags/SSE/Quota | Hidden | Exported |
| Additional real deletions in initial synchronization | Existing behavior | Include all four deletable types |
Old implementations cannot safely consume all newly exposed deletion information. For example, old Quota heal can clear the payload while retaining an already parsed cache value. An instruction to upgrade does not itself isolate this rolling-upgrade window, so the default stays off.
Upgrade every node at every participating site to a build containing the repair, ensure consistent settings within each site, and drain old requests. Then set on consistently and restart. Before a downgrade, first set off and restart every fixed node, then roll back the software. The old software’s original defects return with it.
While off, hidden Tags/SSE/Quota tombstones can cause repeated stale heal RPCs that a fixed receiver rejects. A subsequent heal with zero RPCs is expected only when complete state is visible and stable.
Retained and rejected alternatives
| Decision | Reason |
|---|---|
| Retain one internal six-field helper | The same source-time defect was reproduced across all six; shared ordering prevents entry-point drift while preserving type-specific deletion behavior |
| Keep the whole-bucket lock, persistence fields, and heal interval | They provide atomicity, durable deletion state, and missed-event recovery without another coordination service |
| Do more than replace UTCNow with source time | That alone leaves stale lock-external decisions, invisible deletes, equal-time conflicts, and initialization gaps |
| Do more than export tombstone times | Old receivers and Quota cache handling remain unsafe without controlling the rollout window |
| Do not use deployment ID as a tie-breaker or add an HLC/schema | Existing time and content keys suffice for the bounded contract; cross-site causal ordering is not claimed |
| Do not reject all zero-time typed events | Old Tag heal really omits time; preserve inexpensive protocol compatibility with an explicit limitation |
| Do not impose one deletion rule on all metadata | That would delete Versioning/Object Lock or violate separate Lifecycle/CORS rules |
The initial implementation adds 729 and removes 692 production Go lines, a net increase of 37, chiefly replacing duplicated apply and heal branches. Line count does not establish minimality. Necessity must connect each mechanism to a concrete failure; sufficiency must cover every real entry point; minimality asks which failure returns if a mechanism is removed.
Validation and its limits
The environment is local go1.27.1 darwin/arm64. Four groups of pre-implementation audit tests failed on the unfixed baseline. The resulting regression suite passes on both ObjectLayers; its main entry points are in cmd/site-replication-metadata{,-heal,-gate}_test.go. The baseline audit log retains the original failures alongside the passing existing tests.
| Validation | Observation |
|---|---|
| Source time, four deletions, duplicate/reordered events, queuing before the lock, different-field writes | Regressions and targeted race checks pass |
| Both equal-time arrival orders, deletion priority, Policy keys and negative-set GET, bulk/import | Boundary regressions and supplemental race checks pass |
| Full cmd package and internal/S3 Select race | Full cmd passes on final production code fcbb93e89 (492.776 seconds); internal/S3 Select race passed at 62cf066ff |
| Build, vet, lint, generated files, compatibility checks | Final production build/vet pass; lint reports zero issues at 461e9a721. Generated-file and compatibility checks passed at 62cf066ff. Optional typos is unavailable and skipped according to the Makefile |
| Linux/Darwin/Windows × amd64/arm64 | Six cross-compiles pass at 62cf066ff; this is not runtime acceptance on six platforms |
| Two real site processes, four data directories per site | Initial synchronization preserves Created for six historical live configurations; normal 30-second heal restores consistency after dropping a real PUT’s outbound RPC and injecting reordered events |
| Missed deletions and restart | Four deletion states persist across source-process restart and converge after reconnection |
| Quiescence and diagnostics | Two separate 65-second observations contain no metadata RPC across two normal heal cycles; repeated exceptional events deduplicate by bucket/field/reason |
| Fixed and pinned old implementation together | Tags PUT/DELETE smoke passes with all switches off; this does not prove complete mixed-version correctness |
| Review repairs: creation-time recovery, policy status key, heal diagnostics | Real ObjectLayer creation-time and legacy-order policy tests fail with old production code overlaid and pass after repair. Full cmd, internal packages, S3 Select race, lint, generated files, branding checks, and six cross-compiles were rerun at 62cf066ff |
| Final cleanup regressions | Targeted diagnostic, initial-sync, physical-time boundary, adoption, and CORS race tests pass at fcbb93e89; related race tests pass again after test-style changes in 461e9a721 |
| Two-site acceptance repeated on the final binary | The same run passes on binaries built from clean 62cf066ff and clean fcbb93e89; final 461e9a721 only changes test style and has identical production code |
Local evidence contains baseline failures, test logs, a rerunnable two-site driver, metadata snapshots, and source/binary SHA-256 identities. The first two-site binary reports 01aaef2b5 + dirty; its actual Go build-info and SHA-256 have now been recorded. After review repairs and cleanup, runs were repeated on binaries built from clean 62cf066ff and fcbb93e89. Actual --version, Go build metadata, and SHA-256 identities are retained with the baseline binary built from 5c5765816. Final 461e9a721 differs from fcbb93e89 only in test formatting and equivalent conditional syntax; that diff is recorded separately.
The table above records local tests and isolated-process observations. Remote integration evidence is separate: all nine checks on PR #180 passed, comprising six Go CI jobs, DCO, vulnerability analysis, and release-pipeline validation. The merged main tree was verified to match the PR merge tree that passed these checks. This is not proof of real Linux multi-node cluster behavior, official release artifacts, or production deployment.
Adversarial review record
The plan used actual Claude Code claude-opus-5 --effort max for four rounds. The first two drove corrections to state comparison, committed snapshots, historical baselines, and import boundaries. The last two returned GO_WITH_NONBLOCKING_NOTES with zero pre-implementation blockers. Plan approval is not proof that implementation is correct.
The separate implementation review pinned 1ee64a8d8 and used the same model and effort to inspect the complete diff, production call chains, formal tests, and runtime evidence independently, challenging sufficiency, minimality, rollout safety, and evidence identity. Its verdict was GO_WITH_NONBLOCKING_NOTES: no unconditional blocker, one conditional blocker, and nine further findings. It confirmed the core convergence mechanism and found no counterexample to ordering, deletion, or duplicate suppression within the declared contract. Three findings were real defects the change had introduced and were repaired in 62cf066ff. The subsequent fcbb93e89 closes the diagnostic gap when no valid source exists and adds an initial-sync regression.
| Finding | Assessment | Disposition |
|---|---|---|
| F1: a bucket with no recorded creation time can no longer write any of the six configurations | Confirmed regression, conditionally blocking | Repaired. GetBucketInfo returns the physical probe unchanged for a metadata-free request, as ListBuckets already did; initial synchronization recovers the time and passes it to the bucket creation hook |
| F2: replication status compares statement order while heal compares the canonical key | Confirmed; a permanent false mismatch that heal can never resolve | Repaired. Status compares the same key heal compares; per-site presence counting is unchanged |
| F3: one log key for four heal conditions, at error level for a normal transient | Confirmed | Repaired. Each reason keeps its own key at warning level. Empty baselines and missing buckets stay quiet; invalid existing state is still diagnosed when no valid source exists |
| F4: three user-visible semantic changes not written down | Partly valid | The original reviewed README already documented empty Policy and zero Quota at its end; the first review missed them. The addition covers Policy GET/export ordering and negative-set behavior |
| F5: whether the Policy encoder has unnecessary callers | The second review corrected the first assessment | Comparison and status keys must agree. GET/export/peer need the encoder to read or replicate negative-set policies that can already be persisted. PUT/import normalization is optional for comparison, but removing it adds branches and representation differences, so it stays |
| F6: an orphaned helper and a stale ordering comment | Confirmed nits | Comment corrected. The unused isBucketMetadataEqual and the obsolete test of that helper have been removed |
| F7: with the switch off, missed Tags/SSE/Quota deletions do not converge | A correct reading of the plan’s trade-off | No change; stated in rollout and in the limits below |
| F8: evidence gaps - a stubbed recovery test, no legacy-order policy case, unverifiable binary identity | Confirmed | Recovery now runs on the real ObjectLayer; a permuted legacy policy case was added; the two-site acceptance was repeated on a binary built from the clean final tree, with identities recorded |
| F9: bucket generation conflicts | Declared out of scope, and not a regression | No change; see the limits below |
| F10: adoption moving Created later than a real field time | A coverage gap, not a defect | The case now fixes both sides: preserve historical time, exclude earlier-generation state as a source, and accept a valid adopted-generation input as a target |
A stub is worth calling out separately. The original recovery test injected an object layer whose creation probe returned the expected time, so it passed against code that could never behave that way in production. The replacement stamps the bucket directory on every local drive and drives the real object layer, and it fails on the unrepaired code.
The second review pinned 62cf066ff, again using actual claude-opus-5 --effort max. It returned GO_WITH_NONBLOCKING_NOTES with zero conditional or unconditional blockers. It retraced production paths, checked the author’s F1/F2 reproduction results from formal tests with old production code overlaid, and revised the first review’s assessment of the Policy encoder. The reviewer did not execute the tests.
| Follow-up finding | Final disposition |
|---|---|
| NB-1: physical Created is an approximation | Retain the generation boundary and document directory-mtime limits. A real-drive test fixes the behavior: an earlier peer event is skipped and a local correction succeeds. One source timestamp cannot lower bucket identity |
| NB-2 and NB-8: silence without a source; lost source time in recovery-error logs | Repaired. Invalid existing states remain visible and recovery failures retain the event time. No source means no RPC; empty baselines stay quiet |
| NB-3: outage logs multiply by bucket and field | Accept and document the existing bucket/field/reason granularity; a site-wide log aggregation framework is outside this correctness repair |
| NB-4: draft logs are not product-failure evidence | Mark findings-before.log and findings-after-1.log as superseded fixture failures. Formal tests with old production code overlaid reproduce the physical-time, policy-order, initial-sync, and diagnostic defects separately |
| NB-5 and NB-6: unused helper; incomplete recovery-path description | Remove the helper and both tests that only exercised it. Document initial sync’s persisted recovery and retain real CORS-path tests |
| NB-7: adoption covered only the source side | Add the target side: valid adopted-generation input replaces invalid earlier-generation state |
Diagnostic regressions now use the existing logger target to check warning level, distinct reasons, repeated calls, and zero RPCs. The initial-sync test drives a real source ObjectLayer and the complete outbound sequence; its peer only acknowledges RPCs. That proves outbound content. The real two-process experiment supplies a separate level of evidence, and the two must not be conflated.
The third focused acceptance pinned final production code fcbb93e89 and again returned GO_WITH_NONBLOCKING_NOTES, with zero blockers. It also inspected the test-only style diff in 461e9a721 and confirmed equivalent semantics. Full cmd and lint were still running when the reviewer read their logs; both subsequently exited with code 0. Test formatting caused lint to fail at fcbb93e89 itself; corrected 461e9a721 is the delivery baseline that satisfies the required checks.
Three nonblocking improvements remain outside this repair: heal diagnostics can show a zero source time after decode/parse failure; the initial-sync unit test does not execute the local peer branch and therefore does not prove that recovered Created is persisted locally (that production path was reviewed); and strict system-log capture could need isolation if concurrent background logging is introduced. No additional production accessor or test hook was added for these points. A counterfactual failure proves the assertion actually reached, such as empty-baseline noise. Warning-level and per-reason deduplication assertions pass after repair; that is not a claim that each was separately demonstrated failing on old code.
Operational boundaries that remain
- Historical timestamp pollution is not reconstructable. Arrival-time replacement and old untimestamped events have lost source facts. Installing the repair cannot recover their true historical order. Inspect every site and resubmit the intended configuration or deletion at the authoritative site.
- Zero-time typed events remain compatible. They use monotonic local time and emit
legacy-zero; the existing zero-time constraint for bulk remains. These events are outside the timestamped-source convergence guarantee. - Bucket identity conflicts are not merged automatically. Resolve generation differences first. Events before target Created are not applied; unknown creation time is recovered only from a real physical bucket. Unknown or missing buckets are not written. An existing bucket whose physical creation time is still zero produces an InternalError (500) and aborts initial synchronization; a missing bucket instead retains NoSuchBucket (404). An invalid existing field can also block a typed write with InternalError; heal may replace it using a valid peer source. The recovered value is a physical approximation from bucket-directory modification time. It can change with top-level entries, differ across drives, and be later than actual creation. Older source events are still skipped; one event is insufficient to lower the bucket identity. A successful configuration write or initial site sync records the recovered value. Until then, status reports the unknown time and periodic healing skips the bucket in both directions.
- A physical clock is not a causal clock. Local correction can advance beyond an already known future field time, but cannot infer the business intent of all concurrent writes.
- Diagnostics are bounded; success does not mean applied.
legacy-zero,before-created,indeterminate,unreachable, andpeer-errorreuse LogOnceIf with stable keys and error text, details in attributes, and existing hourly cleanup. Each reason keeps its own key, so one condition cannot deduplicate another away. Empty baselines and peers that do not yet have the bucket stay quiet. Invalid existing states emitindeterminateeven when no valid source can be selected, without causing an RPC. Normal duplicates and older events remain quiet. This is a per-bucket/field/reason bound: both unreachable-peer warnings and indeterminate-state warnings can grow with bucket and populated-field count, not a fixed site-wide limit. - Code, integration, and release require separate acceptance. Issue state, Server version, images, packages, published documentation, and production settings each need their own evidence. The main-branch merge recorded here does not establish that release artifacts or production installations have been upgraded.
4.2 - An Unsigned Header Is Not Part of the Request
This record describes the unsigned-header coverage repair committed to SILO as 123325430 and merged through PR #173, tracked as SN-2026-011. It was reported by Oren Yomtov against a released build and reproduced locally on both signature paths.
Status on 2026-09-11: the original repair is pushed and merged through PR #173. The follow-up signing and payload-verification fixes described below are also merged through PR #177, with all eight PR checks passing. Source validation and published releases are separate: the currently published September 3 Server release does not contain these fixes.
Scope: SigV4 header coverage, consistent policy inputs and body-checksum verification. S3 field names, object and bucket metadata formats, replication protocols, encryption formats and client commands are unchanged.
Security property for ordinary signed and presigned SigV4: unsigned client-suppliedx-amz-*operation headers cannot change an authorized request; policy evaluation and body verification use the effective signed inputs.
Too Long; Didn’t Read (TL;DR)
A presigned PUT URL can sign only the host header. SILO confirmed that each header named in the signed-headers list had arrived, but it never walked the headers that actually arrived, so an x-amz-* header outside that list was accepted and used. cmd/api-router.go routes any PUT carrying x-amz-copy-source to CopyObjectHandler on that header alone. Together these turned a write grant for one object into a server-side copy that reads any object the signing key can reach, executed as the signer — a confused deputy. The Authorization-header path behaved the same way when the header was left out of SignedHeaders.
The repair states one invariant:
This matches AWS S3, which refuses the same request with AccessDenied (“There were headers present in the request which were not signed”). The affected signing code was inherited from upstream minio/minio. The released SILO baseline predates this repair; source provenance alone does not establish the status of every upstream build or other fork.
Failure: the coverage gap
extractSignedHeaders in cmd/signature-v4-utils.go iterates the signed-headers list and, for each name, pulls the value from the request (or the query string). It proves that every promised header is present. It never asks the opposite question — is every x-amz-* header that arrived actually in the list?
The one place that walked the arriving headers, checkMetaHeaders, matched only the X-Amz-Meta- prefix and was called only from the presigned path (doesPresignedSignatureMatch). The Authorization-header verifier (doesSignatureMatch) called nothing equivalent. So an unsigned x-amz-copy-source — or any other operation-shaping x-amz-* header — sailed through on both paths:
Reproduced locally on RELEASE-style builds: the control PUT returns 200 with an empty body; the same URL plus the one unsigned header returns 200 with a CopyObjectResult whose ETag is the md5 of the victim object, and the destination reads back the victim’s bytes. Where the destination bucket already allows anonymous GetObject, the copied private bytes are then readable with no credentials at all.
Provenance
The gap is inherited from upstream MinIO; SILO did not introduce it. The SigV4 verifier in cmd/signature-v4-utils.go and the Authorization-header path doesSignatureMatch in cmd/signature-v4.go are original MinIO code dating to 2016, and the header-driven CopyObject dispatch in cmd/api-router.go traces to 2019. The only routine that ever walked the arriving headers, checkMetaHeaders, was added upstream on 2023-07-27 in minio/minio#17737 (535f97ba6). Upstream therefore recognized the class — an unsigned header must match the signed set — but scoped the check to the X-Amz-Meta- prefix and to the presigned path, leaving x-amz-copy-source and the whole Authorization-header path uncovered. The inherited verifier predates the SILO fork.
Before the unsigned-header repair, SILO’s change to cmd/signature-v4-utils.go was the one-line dependency-path migration in 9b11dc946, moving the policy import to pgsty/silo-pkg/v3. The vulnerable verification behavior came from upstream. The original repair (123325430) and the follow-ups in PR #177 change that boundary. The vulnerable code predates SILO’s fork baseline, the upstream 2025-12-03 maintenance-mode commit from which the first SILO release was cut.
This record establishes the repair in the maintained SILO source graph. It does not claim that SILO is the only implementation with a fix, or establish the current maintenance status of other projects.
The repair
checkMetaHeaders becomes checkUnsignedHeaders, is broadened from the X-Amz-Meta- prefix to all of X-Amz-, and is called on both the presigned and Authorization-header paths. A header that is not covered by the signed set is refused with ErrUnsignedHeaders before any handler logic runs.
Four decisions shaped the exact boundary. Each had a plausible alternative that was rejected for a concrete reason.
Membership, not value equality
The inherited check compared signedHeadersMap.Get(k) == val[0]. For a header absent from the signed map, Get returns the empty string, so a header whose first value is empty compared equal and passed. A multi-value header such as X-Amz-Copy-Source: ["", "/src/secret"] could therefore smuggle an unsigned copy-source past a value-equality check. The repair tests membership in the signed set instead. A signed header’s value is already bound by the signature, so value equality was never the property that mattered; presence in the list is.
Exempt X-Amz-Content-Sha256
X-Amz-Content-Sha256 can be omitted from SignedHeaders because the effective payload hash is bound separately in the canonical request. For presigned requests, the query value takes precedence, with a header fallback when the query value is absent. An explicit UNSIGNED-PAYLOAD remains valid. PR #177 aligns the policy condition with this effective value while preserving header-presence semantics, and checks header-only presigned body hashes in the generic authentication path as well as upload paths. The exception does not permit policy evaluation or body verification to use a different value.
Derive signature age from the signed date
The original repair exempted an internal x-amz-signature-age scratch header written after verification. That was too late for PUT and UploadPart authorization, which runs before signature verification. PR #177 instead derives s3:signatureAge directly from the signed X-Amz-Date, and removes the scratch header, its constant and its exemption. A forged date fails signature verification; an unsigned client header under the old name is rejected. Verification remains idempotent without mutating request headers.
Inject X-Amz-Tagging after authentication
PutObjectTaggingHandler derives an X-Amz-Tagging header from the request body so that policy conditions can read it, and it previously did so before authenticateRequest. With the broadened check, that server-synthesized header — which the client never signs — would be refused as unsigned. The injection now happens after signature verification and before authorization, which still has it for policy conditions. Rejected alternative: blanket-exempt X-Amz-Tagging the way content-sha256 is exempt. That would let a client set object tags through an unsigned header on any signed or presigned write, reopening a smaller version of the same class of bug.
Scope across signature modes
- Authorization-header (signed) and presigned SigV4: both now enforced. These are the reachable paths.
- Streaming SigV4: the seed verifier does not call
checkUnsignedHeaders. The copy handlers reject streaming authentication through their ordinary authentication dispatch, but that does not establish complete header coverage for streaming PUT/UploadPart paths. No dedicated streaming-copy rejection test is claimed here. A universal coverage guarantee requires separate implementation and regression evidence. - SigV2: unaffected. V2 canonicalization folds the
x-amz-*headers into the string-to-sign by construction, so an addedx-amz-*header changes the computed signature and is rejected as a signature mismatch.
Status code: 400 versus 403
AWS returns 403 Forbidden for an unsigned header; SILO returns 400 AccessDenied (ErrUnsignedHeaders), inherited from upstream. The attack is refused either way, and the error Code string is identical; only the HTTP status differs. Raising it to 403 is a one-line change to cmd/api-errors.go that also shifts the pre-existing meta-header rejection. It is left as a deliberate, reversible election rather than folded silently into a security fix, because it is a behavior change for the existing unsigned-meta-header path and is not required to close the vulnerability.
Tests
Several existing tests built a signed request and then set x-amz-copy-source, x-amz-copy-source-range, or x-amz-metadata-directive after signing — that is, they depended on the very behavior this fix removes. They now re-sign with signRequestV4 after setting those headers, as required for these operation-shaping headers by the repaired verifier. signRequestV4 excludes the Authorization header from its own signed set, so re-signing is safe. Current coverage includes checkUnsignedHeaders unit cases for empty first values, the payload-hash exception and rejection of the obsolete unsigned signature-age header and TestPresignedVerifyIdempotent, which verifies the same presigned request twice.
Evidence
- A built server reproduced the confused deputy on both the presigned and Authorization-header paths, then refused both after the fix while the control
PUT, a realminio-goCopyObject,PutObjectwith user metadata and tags, and body-basedPutObjectTaggingall continued to work. go test ./cmd/passes on the fix tree;gofmt,gofumpt, andvetare clean.- Adversarial review (round one) independently surfaced three defects in the first draft — non-idempotent verification via the scratch header, the empty-first-value bypass, and over-rejection of an unsigned
x-amz-content-sha256— each of which is addressed above and confirmed by re-running the reviewer’s own adversarial test suite against the final tree. - Adversarial review (round two, against the committed fix) found no regression and confirmed the re-signed tests keep their original intent: an invalid access key still returns
InvalidAccessKeyId, and a wrong SSE-C key still returns403after the signature validates. It surfaced three adjacent, pre-existing gaps that also fail on the parent commit and are out of this change’s scope; they are recorded under follow-ups below.
Compatibility and operations
- Ordinary clients: no request change. Conforming ordinary signers include the nonexempt
x-amz-*headers they send. Custom signers must verify this contract; streaming modes have the separate boundary above. - Unsigned
x-amz-*headers: now refused withAccessDenied, as on AWS. A client that added such a header without signing it was already outside the SigV4 contract. - Rolling upgrade: wire and storage formats are unchanged. Upgraded nodes enforce the boundary; nodes still running an older build remain exposed until upgraded, so behavior can differ by node during the rolling window.
- Rollback: data written by the fixed version stays readable by the previous version, but rollback reopens the confused deputy.
Residual risks and follow-ups
- Release delivery: a source fix and a public engineering record do not establish that a published binary or image contains the fix. Verify the selected release and artifact separately.
- CVE: the reporter requested one; the finding carries the stable fork-local
SN-2026-011identifier until a CVE is assigned. - Status code election: the
400-versus-403choice above is open. - Adjacent signing fixes: PR #177 addresses repeated copy-source ambiguity, signature-age authorization ordering, and the effective payload-hash policy value. It also closes the separately reproduced header-only presigned body-checksum gap. The regression set covers signed and presigned requests, policy enforcement before upload verification, and real HTTP bucket-policy tampering. These follow-ups are distinct from the original
SN-2026-011finding; their merge and release status is recorded above. - The general question: this repair covers
x-amz-*request headers. Any future control that lets request syntax select an operation must answer the same question this one did — is this value covered by the signature before it is allowed to mean anything? The repeated-header gap above is the same question in a different guise: the value the signature binds and the value the handler consumes must be the one and the same.
Conclusion
The signature is the request. Everything an x-amz-* header claims is a claim until the signature covers it:
Confirming that the promised headers arrived is not the same as confirming that the arrived headers were promised. Refuse any unsigned
x-amz-*header before the handler runs, on the ordinary signed and presigned paths covered by this repair. Streaming coverage remains a separate boundary.
The ledger separately tracks the payload-verification repair as SN-2026-012. The signing change alone does not change storage formats, but the same main candidate also contains IAM changes requiring coordinated upgrade. Do not use this record as approval for a rolling upgrade of that entire candidate.
4.3 - Durable IAM Revocations
Source status, 2026-09-16: #191 and #192 are merged into main, but are absent from published Server 20260903. They change persistent IAM state and require a coordinated upgrade. Follow the upgrade and recovery runbook and SN-2026-013.
SILO retains the version of a deleted IAM record so an offline site cannot restore an older identity or grant when it reconnects. This covers built-in users, service accounts, groups, policy documents, and policy mappings in their actual user/STS-parent/group namespaces. It also revokes the deleted built-in user’s older service accounts, STS credentials and group grants across deliberate same-name recreation.
Ordering and persistence
Deletion records occupy the original IAM configuration paths. They contain the
originating timestamp, Deleted, and, where required, RevokedBefore; identity
deletion records contain no secret key or session token. Normal IAM listings and
authorization hide deleted records. Object storage and etcd both serialize each
path’s version comparison and write with a distributed lock. The source timestamp
is persisted without replacing it with the receiving node’s clock.
Older events cannot overwrite a newer revision. At an identical timestamp, a deletion wins over a live record. A local deliberate recreation receives a version newer than the stored deletion. An older user/group deletion arriving after recreation retains its revocation boundary while preserving the newer live record. Receiving an already-applied tombstone does not rewrite or advance it. These rules also apply after cold loading persistent IAM state.
The user or group revision is the commit point of deletion. Cleanup of mappings and children follows that commit; cleanup failure cannot undo it. The API returns the cleanup error and still notifies sibling nodes to reload the committed state. Such an error does not mean the identity is still active. Retry the intended revocation only while IAM writes and same-name recreation are paused. When the API returns a committed-cleanup error, the admin handler does not send its immediate cross-site hook; cross-site propagation relies on the normal deletion-healing retry. Sibling deletion notifications reload current shared state, so a delayed notification cannot delete a subsequently recreated identity. Etcd siblings also receive persistent changes through watches. Failed notifications/watches remain eventual propagation, not a distributed instantaneous revocation transaction.
Keep node clocks synchronized and monitor offsets. Ordering uses source wall clock timestamps, with monotonic advancement for local writes to the same path. It does not establish causal order between concurrent writers at different sites or resolve conflicting live updates with identical timestamps deterministically.
Users, child credentials and groups
A recreated user retains RevokedBefore. New service accounts and built-in STS
issuances carry a signed siloParentRevocation claim identifying the parent
boundary known at issuance. Editing or replaying an old child does not update
this claim. Old children remain invalid even when their own update timestamp is
newer than the parent deletion; children issued for the recreated parent remain
valid. Claims are read from verified tokens on credential load/write.
A newer replicated service-account snapshot can replace an older service with the same access key, including a deliberate change of owner or secret. Local duplicate creates remain rejected. Snapshots preserve disabled status and the service’s own revocation boundary, so earlier mappings cannot attach to the new service. Equal-version retries reload the committed identity without rewriting it. Periodic live healing compares source versions even when the public status summary is unchanged, and includes disabled identities as healing sources. Service snapshots also preserve their absolute expiration. The receiving site does not reapply the local minimum lifetime for a newly issued credential. A newer already-expired snapshot still supersedes the old key, is denied by authentication, and is collected into a durable service tombstone by normal loading. Cache/claims loading failures are returned for retry, not acknowledged; after a committed replacement the stale cached secret is evicted immediately. Collisions with an existing built-in or cached STS identity report an error and require an explicit administrative resolution; replication cannot change its credential kind. Concurrent conflicting service updates with exactly the same timestamp can retain different winners at different sites; the status summary does not resolve that case.
Each group member has its own MemberGrants timestamp. Changing another member
or the group’s enabled status does not reissue everyone else’s grants. Effective
membership requires the grant to be newer than both the user’s and the group’s
retained boundaries. Listings and policy evaluation use the same effective
membership. Peer snapshots preserve grant times, including unknown legacy grant
times; they cannot treat a recent snapshot time as a fresh grant to a revoked
identity. A new explicit administrative group grant can restore access.
This does not implement a general conflict-resolution protocol for all group membership edits. In particular, the inherited live-group snapshot merge adds members and does not reconcile a missed ordinary member removal. Removing a member from a live group during a site outage is a separate known limitation; do not infer that this change resolves it. User/group deletion boundaries and same-name recreation are covered here.
Retention and expiration
Permanent identities, groups, policy documents and mappings have no automatic tombstone TTL. A disconnected peer or an old backup may return arbitrarily late. Successful replay acknowledgements are an optimization, not permission to garbage-collect this history.
Natural expiration of an immutable STS token physically removes its token-key record and any legacy token-key mapping, without generating a permanent tombstone. An early STS revocation is retained until that token’s expiration plus the existing clock-skew allowance. Replaying the same revoked token with a later event timestamp cannot recreate it. Etcd uses an expiration lease; object storage collects expired STS tombstones during its existing credential loading/purge. A record with unknown expiration is retained conservatively. Cleanup writes are best effort and use a short lock budget. If one fails, that load stops optional reclamation, reports the error and still loads healthy users; expired credentials stay denied and retain their existing durable version. The next load retries. Healthy cleanup has no fixed record quota. The reusable STS parent policy mapping is not assigned the token’s TTL by deletion cleanup. External-IDP disablement is an early revocation, not natural token expiration, and its cached STS and service accounts are included in cleanup. Expiring service accounts retain a durable revision because their access keys are reusable and an older version might have no expiration.
Direct per-token RevokeTokens delivery between sites is not a new guarantee of
this change. The guarantee for built-in parent deletion follows from the durable
parent boundary, including children not currently present in the deleting node’s
cache.
Healing, failures and operational cost
Each process starts one healing loop. Losing its distributed leadership lease pauses work until leadership is reacquired; it does not permanently terminate healing. Configuration reloads do not create extra loops. The 30-second interval starts after leadership is acquired and after each completed pass. Initial lock retries and endpoint recovery can add further delay; it is not a convergence SLA.
The normal IAM loaders maintain an in-memory index of deletion records and
retained boundaries, without secrets. Healing uses this index; it does not add a
second full walk of config/iam/ every cycle. Existing full IAM loading still
scans persistent records, including tombstones, at startup and on refresh.
Each peer receives batches of at most 128 records. The sender remembers which path/version each peer acknowledged. Unrelated new changes at either site do not reset that progress. A failed batch remains pending while later independent batches can progress; a lost response may cause safe idempotent replay. A pass has a bounded duration, and its successful acknowledgements survive that timeout. The protocol reports each node name and process instance. Switching between known node instances behind a load balancer preserves acknowledgements. A new node instance conservatively invalidates prior acknowledgements once; repeated switches among those known instances do not reset progress. Restore persistent state only with the affected processes stopped, so a restore cannot reuse an old process acknowledgement.
Steady-state healing still traverses/sorts the retained in-memory set and checks the peer’s protocol status. It suppresses repeated deletion PUTs once acknowledged. Memory use scales with retained paths and peers; startup storage reads scale with history. This release does not provide general history compaction. After upgrading, older live records whose receiving sites originally assigned different timestamps can require an initial reconciliation wave. Allow for its storage writes and sibling notifications when planning the maintenance window.
The cluster IAM metrics include revocation_records,
revocation_heal_failures, revocation_heal_duration_millis, and
revocation_heal_last_success_timestamp_seconds. Errors are also logged. A
nominal 30-second scheduler interval is not a convergence deadline: outages,
large backlogs, lock contention and failed requests can require more passes.
Cached credential lookup checks the in-memory parent revision index and performs no additional storage read. STS issuance and cold credential loading still consult the persistent parent revision. Revision I/O and distributed lock waits release the IAM cache lock while a separate local writer mutex preserves write order. These operations have bounded contexts, including etcd lock and lease cleanup.
Protocol and supported upgrade
The server-owned versioned route is
/minio/admin/v3/site-replication/peer/iam-revisions. It carries source versions,
member grant times and distinct user/group revocation items without changing the
admin client SDK or S3 API. Older servers reject this route. The sender reports
the failure and does not fall back to a route that would discard the metadata.
The existing legacy IAM route remains readable for best-effort compatibility;
this does not confer the new guarantees on an older peer.
All participating servers must be upgraded for the guarantee in this document. Mixed old/new nodes sharing an IAM backend and rolling downgrade are unsupported: older binaries do not interpret tombstones or signed parent boundaries correctly. Use a maintenance window for coordinated upgrade:
- Pause IAM changes and isolate any offline site or backup whose state is unknown.
- Back up each site’s complete IAM storage and required encryption material. A live IAM admin export omits deletion history and is not an adequate backup.
- Stop all nodes sharing each site’s IAM backend, replace their binaries, and restart them on the upgraded version. Complete this for every participating site before relying on the new revocation semantics.
- Check IAM loading, site-replication errors and revocation convergence. Verify representative old credentials are denied and deliberately reissued ones work.
- Resolve pre-upgrade revocations explicitly. Absence cannot reconstruct an already-lost deletion version: remove surviving old records on the sites that still have them, and rebuild stale offline peers from approved state before admitting them. Do not reconnect an unknown old snapshot just to discover its deleted credentials.
Credentials issued by an older server for a recreated parent lack the required signed boundary and must be reissued by an upgraded server. Parents without any retained revocation history preserve existing credential behavior.
A pristine built-in policy remains protected from local deletion. If an administrator explicitly overrides that policy and later deletes the override, the durable deletion now suppresses automatic recreation of the built-in policy on reload. This prevents reload from undoing the deletion. Restore the policy by an explicit policy-create operation if desired. Local deletion of a nonexistent policy remains idempotent and does not create a new tombstone; replicated unknown deletions retain their version.
For rollback, stop and isolate the affected sites and assess changes since the backup before restoring compatible state. Restoring an older backup can itself lose later revocations and requires reconciliation/rekeying before access is reopened. Do not delete tombstones online or convert only live IAM records to make an older binary start. Server, client, Console, package and deployment acceptance remain separate delivery gates of the maintained PGSTY stack.
Regression and performance checks
Focused coverage is in iam-revocation_test.go, iam-revision_test.go,
iam-revision-lock_test.go, iam-revision-boundary_test.go,
iam-replication-protocol_test.go, iam-credential-retention_test.go,
iam-peer-reload_test.go, and iam-replay_test.go. Set
SILO_TEST_IAM_REVOCATION_ETCD to a disposable etcd endpoint to include backend
lifecycle/locking/boundary tests; they use isolated key namespaces.
BenchmarkIAMCachedCredential and BenchmarkIAMSetTempUser can be run against
the pre-change source for a comparable local baseline.
BenchmarkIAMRevisionConvergedHealing covers 1,000 and 10,000 retained records;
it measures steady-state index/network work and asserts zero repeated PUTs. Its
fake peer does not measure durable catch-up throughput.
BenchmarkIAMColdLoadExpiredServices measures loading and cleanup with 100 or
1,000 expired reusable credentials; run it with -benchtime=1x. Use actual multi-site
signed S3/STS tests and deployment-specific latency/scale measurements in
addition to these component tests.
Errors and observability
An admin delete can return HTTP 500 after the authoritative revocation has committed but dependent cleanup failed. It is not evidence that the old identity remains usable. Retry the intended revocation while IAM changes and same-name recreation are paused, then verify old and reissued credentials at every site. When a persistent parent revision cannot be read or issuance cannot be committed, STS fails closed through an internal-error path, including STSInternalError; it must not mint a credential by guessing a missing boundary.
Scrape each process’s authenticated /minio/metrics/v3/cluster/iam endpoint:
| Metric | Interpretation |
|---|---|
minio_cluster_iam_revocation_records |
Retained deletion records and parent boundaries in this process’s index |
minio_cluster_iam_revocation_heal_failures |
Failed convergence passes since process start |
minio_cluster_iam_revocation_heal_duration_millis |
Duration of the last pass |
minio_cluster_iam_revocation_heal_last_success_timestamp_seconds |
Unix time of the last successful pass |
The index is per process; summing it across siblings double-counts shared state. Healing requires site replication and a leadership lease. Shared-backend deployments without site replication do not run the site healing pass; zero healing metrics are not evidence of a failure there. A healthy counter on one leader does not prove every peer’s credential checks pass.
Why simpler alternatives fail
Physical deletion alone loses the ordering evidence an offline peer needs. Comparing only the child’s update timestamp is also insufficient: editing an old child after a parent deletion would appear newer and could restore access. The signed issuance boundary remains unchanged by such an edit. Replacing a received source revision with local receive time breaks source order and can turn a retry into a new mutation. The implementation preserves source timestamps and separately advances only deliberate local writes. Tombstone TTL or restoring only live export records discards the very history that prevents replay.
The versioned source and metrics definitions define this contract. This record replaces the former in-repository IAM design; it is not a production recovery certificate.
4.4 - Federated CopyObject: Preserve the Destination Contract
Source status, 2026-09-16: this records the main-branch CopyObject fixes in #157, #159, #163, #177 and #179. They are absent from Server 20260903. The subject is the legacy etcd bucket-federation path that forwards a copy to another deployment as PutObject, not the bucket/site replication scheduler.
Forward bytes once, encrypt at the destination
The proxy reads logical source bytes: it decrypts and decompresses as needed, then declares the logical length to the destination. It must not locally encrypt/compress and ask the remote to do it again. SSE-C reads still require the source key and secure transport. The destination’s explicit SSE headers or its own encryption defaults choose how the new object is stored; the proxy must not inject its local defaults when the caller selected none. SSE-KMS context is forwarded in the expected JSON-object form.
Trusted raw SSE-C replica CopyObject is explicitly rejected with 501 NotImplemented in this federation path, before creating the destination. That combination is not made safe merely by passing replication markers. Ordinary key-authorized SSE-C copies and the dedicated replica path are different operations.
Checksum and metadata rules
Remove every reserved internal metadata prefix, case-insensitively, from the ordinary forwarded write. Forward public metadata and tags through supported fields, not internal storage encoding. The checksum must describe the logical full object actually written:
- A requested algorithm or a multipart-composite source requires a full-object checksum at the destination.
- For a nonempty stream, the client uses a trailing checksum, including
x-amz-trailer; the destination must consume and validate that trailer. - Empty content uses the ordinary checksum header because there is no streamed checksum trailer to carry its digest.
- A stored full-object source checksum can be forwarded as an ordinary checksum header for validation.
- A required remote checksum that is missing, malformed or has a multipart
-Nsuffix is an error; it must not be reported as a valid full-object result.
The remote write may already have committed when a malformed success response is detected. A resulting error does not prove destination absence. Inspect the written version before blindly retrying a versioned copy.
Object Lock is not user metadata
Forward legal hold through the typed LegalHold option so it becomes x-amz-object-lock-legal-hold, not x-amz-meta-*. Retain the full precision of the retention timestamp; routing it through a whole-second conversion would weaken the requested value. Authentication and the destination’s Object Lock rules still apply.
Return the committed write
The response and ObjectCreated:Copy event describe the destination key, logical size, ETag and exact version ID. Modification time is obtained from the destination write, not invented at the proxy or obtained through a later unversioned HEAD that could observe another writer. The internal write-time response is bound to the authenticated federation request. Source-version response headers still identify the selected source when one was provided.
Evidence and deployment boundary
See the handler, write-time transport, and object-copy-federation*_test.go / object-federation-time_test.go. Tests cover ordinary, empty, multipart, compressed and encrypted sources, remote checksum failures, destination defaults, legal hold, response identity and events. These component fixtures do not certify a production etcd federation or every S3-compatible destination.
Verify both forwarding and destination builds from the component matrix. Existing objects are not rewritten by upgrading. Related records cover SSE-C replica integrity and Object Lock ordering.
4.5 - Multi-Pool Object Consistency
Source status, 2026-09-16: #178 repairs multi-pool mutation ordering; #207 extends current-object selection to conditional PUT. These changes are merged into main and absent from Server 20260903. They close the source defects tracked by #133 and #144; this does not establish acceptance of a new release artifact.
One key can have several physical copies
A pool expansion, decommission or interrupted cleanup can leave copies of one key in different pools. A per-set lock does not serialize an operation with a writer choosing another pool. Nor does reading the first copy prove that its ETag, tags or retention represent the logical object.
The relevant failures include a stale ETag satisfying a conditional write or delete, a newer retention/hold disappearing behind older object metadata, and an older copy becoming visible after deleting the selected copy. Reading each pool independently and combining only its success status does not solve these races.
Shared selection and lock boundary
The pool coordinator holds one namespace object lock across selection and mutation. objectPoolInfos reads the addressed version in every pool, including draining pools. Unknown/unreadable state is an error, not evidence that the object is absent. Copies are ordered by modification time with pool index as the equal-time tie-breaker.
An unqualified metadata request first resolves the logical current version, then gathers copies of that version. It must not merge metadata from unrelated object versions. Object Lock retention, legal hold and tags have independent source timestamps; mergedPoolObjectInfo combines those fields by their own order rather than treating the object’s modification time as every field’s revision. An ordered removal is state too.
Metadata writers, relevant object writers and healing use the same coordinating lock. The lock is not a transaction that can undo completed disk writes on every pool after a later failure.
Conditions and deletion
For conditional DELETE, evaluate once before cleanup and clear the callback before invoking lower pools. An explicit versionId compares that version. For conditional PUT, select the latest logical representation across eligible pools while holding the outer lock, including the delete-marker state, before installing the replacement. #207 addresses the case where a stale destination-pool copy made If-Match/If-None-Match disagree with the current object.
Cleanup and metadata propagation can still partially modify physical copies before a storage failure is reported. A quorum/cleanup failure must not be interpreted as an unchanged object or as proof every old copy disappeared. Verify state and retry after recovery. Batch XML ETag conditions remain unsupported; see conditional DELETE.
Object Lock and replication
Replica writes reconcile destination retention and legal-hold state against authoritative same-version copies across pools. A stale incoming replica must not shorten a newer retention or turn off a newer hold. Tags similarly travel with their timestamp, including an empty deletion value. See Object Lock ordering and replicated tags.
Evidence and operator impact
The coordinator implementation, pool entry points, and the two merged PRs identify the source contract. Coverage includes multiple pools, stale copies, delete markers, explicit versions, concurrent writes and failure paths. It is not a claim of atomic rollback or arbitrary fault tolerance.
Upgrade all participants before depending on the shared-lock guarantees. Inventory historical copies, tags and Object Lock state when a deployment may already have encountered the defect; the source repair does not prove historical cleanup. Use the replica audit runbook and component matrix. The multi-pool repair is separate from the default limitations and scan cost of multipart upload listing.
4.6 - Object Lock Replication Ordering
Release boundary, 2026-09-16: f4c1286c9 shipped in Server 20260903. The later timestamp-only removal, SSE-C retransmission and cross-pool repairs in #129, #134 and #178 are on main and not in that release. The advisory ledger records the original ordering defect.
Preserve the old state before rebuilding metadata
A replica COPY used to rebuild metadata from the incoming request before comparing retention and legal-hold timestamps. The old timestamps were therefore lost, so a stale request could appear authoritative. The legal-hold timestamp also went into the retention timestamp key. The first fix captures the stored state before rebuilding the destination map and keeps each field’s revision under its own key.
Retention and legal hold are independent registers. A later retention does not make an older legal hold authoritative, and the object’s modification time is not a substitute for either field’s revision. Apply an incoming field only when its timestamp is newer than that field’s stored timestamp; retain the stored value for a stale or equal update.
Removal is an ordered value
Removing retention or a hold can leave no live value but still carries a source timestamp. Dropping that timestamp would let a delayed old value return. The later repair recognizes timestamp-only removals during receive, comparison and resend. An absent value with no ordering evidence is different from a recorded removal.
The receiver must preserve these distinctions through ordinary metadata copies and full SSE-C replica retransmission. A retransmitted object cannot erase a newer destination hold or resurrect an older retention just because its body is being rewritten.
Cross-pool authority
If one version has copies in multiple pools, one local set cannot decide the latest field state. The multi-pool coordinator gathers same-version copies under the shared object lock and merges retention and hold independently by their timestamps. Unknown pool state is not an empty value. This closes the source-level boundary originally left open as #133.
Authorization and limitations
Ordering does not grant permission to change Object Lock. The replication trust checks and the relevant S3/admin permissions still apply. It is not a new user API for bypassing governance or compliance retention. It also does not create causal ordering between unsynchronized source clocks or a distributed rollback after partial storage failure.
A historical stale update may already have changed stored state. Upgrading only prevents the repaired paths from accepting the same error again; it does not reconstruct missing retention history. Verify exact-version retention and legal hold at each site against authoritative records before declaring recovery.
Evidence
See the Object Lock merge implementation, the replica write paths in erasure-object.go, and the linked PRs. The tested cases include stale/newer field updates, empty-value removals, distinct field timestamps, SSE-C retransmission and multiple pools. These source tests do not prove every historical replica has converged.
Use the replica audit runbook, SSE-C integrity record and release matrix together. Upgrade all participating nodes before relying on the combined ordering contract.
4.7 - SSE-C Replica Integrity
Source status, 2026-09-16: the repairs in #122, #123, #124, #126 and #134 are on main, not published Server 20260903. Earlier zero-byte/read-attribute authentication and destination-key checksum repairs did ship in 20260903. Do not treat all SSE-C fixes as one release.
Ciphertext and trust
An ordinary SSE-C request supplies the customer key and operates on plaintext. An authorized replica transfer can carry raw ciphertext with the sealed object-key metadata needed to preserve the original object. The destination must store those bytes verbatim; encrypting them again produces an unreadable double-encrypted object. The internal marker alone is not authorization: the exact replication marker and the required replication permission remain mandatory.
This path does not reveal the customer key or make a normal keyless read permissible. Ordinary GET/HEAD and GetObjectAttributes keep their key/permission checks. Federated raw SSE-C replica COPY is explicitly unsupported.
Multipart sizes and checksums
Encrypted part size and logical plaintext part size are different. Replica multipart records preserve the actual logical size of each part so partNumber reads, ranges and GetObjectAttributes agree with the source after overwrite. Checksums must retain the correct encryption context and logical meaning; a checksum response cannot be decrypted with the source key after a copy committed under a different destination key.
For ordinary SSE-C key rotation, any explicitly requested checksum algorithm forces a complete rewrite on current main, even if it names the existing algorithm. A multipart source becomes a single-part destination; the ETag can change and replication retransmits object bytes. An eligible metadata-only rotation without that request preserves the prior checksum state, including absence. See the operator procedure.
Retransmission and Object Lock
A keyless target HEAD can report an existing SSE-C object as inaccessible rather than absent. The sender must distinguish this from NoSuchKey; it cannot assume an ordinary metadata-only COPY will repair the replica. Existing SSE-C replicas use object retransmission. A destination with undecodable old replica state can be replaced through the repaired retransmit path, while preserving the newer destination Object Lock state and ordering removal timestamps correctly.
A failed old replica is not automatically certified repaired after a Server upgrade. Re-read the exact source and replica versions with approved key access, compare the logical bytes and part boundaries, and check retention/legal hold separately. Never infer integrity from a successful metadata-only HEAD alone.
Compression and historical objects
Current main excludes every new SSE-C write from compression, including normal PUT, multipart initiation, COPY and Snowball. SSE-S3 and SSE-KMS continue to follow their encrypted-compression setting. This avoids a replica format that transports ciphertext without the required compression metadata.
Historical compressed SSE-C objects are not automatically rewritten. Preserve their keys and exact version identities, inventory affected data, and rehearse a supported rewrite/recovery path. The change is preventive, not a background migration or a promise that arbitrary old ciphertext can be recovered.
Verification boundary
The compression decision, replication-trust-ssec-replica_test.go, erasure-multipart-ssec-replica_test.go, replication-ssec-retransmit_test.go and compression-ssec_test.go preserve the relevant source and regression contracts. Tests include incorrect-key/unauthorized controls, multipart byte comparisons and retransmission cases. They do not replace a deployment’s historical-state inventory.
See Object Lock ordering, multi-pool consistency and replica recovery. Upgrade every participating Server before relying on the combined behavior.
4.8 - Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation
This page records the analysis, design choices, review, and implementation of #153, #152, and #137. They belong to the same replication reliability series, but affect operation classification, recovery visibility, and task lifecycle respectively. One general retry patch cannot repair all three.
As of 2026-09-09: PR #162 is merged as
d1105bbb, and all three issues are closed. All eight checks on the tested PR head, followed by main Go CI and VulnCheck, passed.
Second round, 2026-09-16: PR #196 (fix0c61128d2, verification recordaea3882c9) repaired the replication worker’s own delete exits and the persisted MRF marker-recovery path — a different surface from the first round. It is on main and not in Server 20260903; see the second-round section.
Review: the plan was discussed with the installed Claude Code Fable 5.1 Max, followed by a review of the implementation. The final verdict was GO.
Delivery boundary: this work completed code, tests, and main integration. It did not create a Server tag or formal release, and does not establish that existing packages, images, or production deployments contain the fixes.
Overall decision and the surrounding series
The selection rule was to repair a reproduced invariant at the smallest boundary that owns it, retain existing recovery mechanisms, and use deterministic tests to prove that work can finish. Additional complexity needs a concrete counterexample.
| Issue | Confirmed defect in current SILO | Selected repair |
|---|---|---|
| #153: delete-marker purge | Single-object DELETE classified a permanent deletion as marker replication; the remote marker disappeared while source purge remained PENDING | Classify by purge status, matching bulk delete, scanner/heal, and resync |
| #152: invisible MRF drops | The queue was already bounded, but internal drop counters did not reach administration or monitoring; object and delete worker arguments were reversed | Expose existing counters, warn at actual drop sites with deduplication, and align worker routing |
| #137: unreliable resync cancellation | One shared token could not stop multiple runs; blocking phases missed cancellation; stale runs could overwrite terminal state | Give each run an owned context, cancel by resync ID, and constrain registration, finalization, and state updates |
The surrounding series had already separated three concepts that are easy to confuse:
- #136 / PR #138 repaired counter completeness: receive and apply the final result before persisting terminal state, without waiting for the one-minute periodic flush.
- #139 repaired outcome accuracy: an existing destination object does not prove that this update succeeded. Success and failure must come from the target’s actual replication result.
- #137 repairs cancellation and resource lifecycle: the task must stop, its walker, workers, and result consumer must exit, and an old run must not pollute a new run’s state. This change preserves the first two contracts rather than introducing another accounting mechanism.
Authorization to use internal replication semantics belongs to the earlier CORS and replication trust record. Authoritative Object Lock state across pools remains tracked separately in #133, open at the original record date and subsequently repaired on main by #178 (not in Server 20260903); see multi-pool consistency. Closing these three issues does not mean every replication concern is resolved.
The release-gating target is the maintained pgsty/silo stack with Console, mcli, and silo-pkg. Compatibility with upstream MinIO/MC remains best effort. Neither a mechanism proposed for upstream nor an external experiment can automatically be treated as an observation on current SILO.
#153: distinguish marker replication from permanent version deletion
The report and the reproduction differ
The original issue described a sustained HTTP 405 storm involving ILM and replication, and proposed treating every delete-marker 405 probe as completion. Current SILO source and measurements do not justify adopting that explanation and patch directly.
The baseline was 450dcb848. The experiment used two locally built SILO servers, separate disposable data directories, and maintained mcli / minio-go clients. A critical observation was that mcli uses bulk DELETE even for a single key; that route was already correct. A direct single-object S3 DELETE was therefore necessary to exercise the faulty entry point.
| Observation | Single-object DELETE before the fix | After the fix |
|---|---|---|
| Matching delete-marker version at the target | Removed | Removed |
Purge state for that version in source xl.meta |
Still PENDING, although ordinary replication status was complete | Cleanup completed on the first replication attempt |
| Original data version | Retained | Retained |
| Scanner needed to finish this cleanup | Required later recovery | No scanner assistance needed in this experiment |
Checking only that the remote version disappeared would falsely declare success. Source metadata must be inspected too. The reproduction establishes incorrect initial purge classification and completion state, plus an unnecessary probe. It did not reproduce the external report’s sustained 405 storm or request-volume figures. Existing scanner/heal and resync producers already select the proper purge path; they cannot be described as necessarily repeating that erroneous probe every cycle.
Why one classification condition is sufficient
When deleting an existing marker, the object layer can return both DeleteMarker=true and a nonempty VersionPurgeStatus. The former describes the version being operated on; the latter describes the operation now required. They are compatible facts.
The old single-delete handler checked only DeleteMarker and populated DeleteMarkerVersionID. Completion then updated ordinary ReplicationStatus instead of completing the purge. The final condition is:
This matches the classification already used by other producers. The fix belongs in the DeleteObject handler. It requires no storage-format change, relaxation of replica deletion protection, or new recovery task. Legacy PENDING entries remain recoverable through the existing scanner/heal purge path.
Why a 405 is not unconditional success
A 405 from a versioned HEAD can establish that the marker exists at the target. For replicating a delete marker, that can mean idempotent completion. For permanently deleting that version, it means there is still work to do.
Writing VersionPurgeComplete merely because HEAD returned 405 could let the source remove its metadata while leaving the unwanted target version behind. The implementation retains existing 405 semantics. Real remote failures such as 403, 405, and 503 must not be disguised as successful permanent deletion.
Source history traces marker-based classification to an upstream 2020-11-19 commit. The 2023-07-10 optimization changed scheduling from dsc.ReplicateAny() to the returned object’s replication/PENDING purge state while retaining classification based only on DeleteMarker. At this location, the 2025-04-02 commit merely moved Pending to replication.VersionPurgePending; it did not first introduce that scheduling condition. This identifies lineage, not a bisect across every historical release, and does not establish when the entire externally reported storm was introduced.
#152: expose drops from an already bounded queue
MRF, or Most Recent Failures, records recent failed replication work for background processing. It already had limits: mrfSaveCh has capacity 100000, and mrfRetryLimit is 3. The code drops a queue entry when RetryCount > mrfRetryLimit or the save channel is full.
The source object and its pending replication state remain. Dropping a queue entry does not mean losing source data. However, subsequent repair depends on the scanner; a prompt retry can turn into a wait for a scan, so this mechanism does not establish a fixed recovery deadline.
The actual defect was that TotalDroppedCount / TotalDroppedBytes increased internally but were omitted from both public statistics snapshots and from Prometheus v2/v3. A deterministic capacity-one fixture demonstrates the failure: one entry is admitted, a 20-byte overflow entry and a 30-byte retry-exhausted entry are dropped, and internal totals become 2 / 50 while administration reports 0 / 0.
The selected minimum change
- Atomically read the existing counters into both administration snapshots.
- Register and load cumulative counters in metrics v2 and v3, and document their meaning.
- Warn at the two actual drop sites: retry exhaustion and a full MRF channel. Fixed messages and deduplication keys prevent changing counter values from defeating deduplication. Handing ordinary worker overflow to MRF is not itself a drop, so it does not gain a warning here.
- Route ordinary objects, healing, and deletion consistently by
(bucket, objectName)to restore worker affinity for the same object.
| Interface | New metric |
|---|---|
| Prometheus v2 | minio_node_replication_mrf_dropped_operations_total |
| Prometheus v2 | minio_node_replication_mrf_dropped_bytes_total |
| Prometheus v3 | minio_replication_mrf_dropped_operations_total |
| Prometheus v3 | minio_replication_mrf_dropped_bytes_total |
These counters accumulate since Server startup and reset on restart. Operations count entries, not unique objects; one object can contribute repeatedly. Bytes cover known sizes, with deletion entries contributing zero bytes. They measure neither data loss nor the complete backlog. Operators should examine increases alongside replication backlog and target health.
This work does not enlarge the queue, increase retry limits, add a persistent retry scheduler, or introduce another backoff framework. Existing limits continue to bound memory use, and the scanner remains the eventual repair mechanism. The repair makes an invisible condition observable; it does not promise a recovery deadline under arbitrary failures.
#137: make cancellation part of a run’s lifecycle
One token cannot cancel a group
Previously, cancellation placed one unkeyed token into a shared channel. One site resync can cover several buckets, with up to 10 concurrent bucket runs, while dispatchers and workers compete to consume the token. Only one bucket might stop; an unrelated task might consume it; or a leftover token might affect a later task.
Two blocking phases had independent defects: a bare receive from Walk output did not observe cancellation, and sending to a full worker channel did not observe it either. Once a worker exited, the dispatcher could block forever on a channel with no receiver. Walk also inherited the parent context, so an early function return could not stop its own walker.
State handling had related faults: site updateState modified a local value without writing it back to the map; bucket Canceled handling was incomplete; and an old finalizer could overwrite canceled state with Completed.
Registration, cancellation, and status share a write boundary
Each resyncBucket creates an owned context.WithCancelCause and registers before waiting for a concurrency slot. Registration and cancelResyncID share the resyncer’s state lock:
- Registration validates that the target still exists and the resync ID still matches, and reads its current cancellation state.
- Cancellation marks matching Pending / Started states Canceled before canceling all matching registered contexts.
- A registered run still waiting for a slot receives cancellation; a run registered after cancellation sees the canceled state.
- Walk, dispatch, workers, and result sends observe that run’s context. Unrelated IDs are unaffected, and there is no buffered token for a later task to consume.
A dedicated operation mutex serializes site start/cancel configuration setup. Canceling running work still executes if part of the target-configuration loop fails. Bucket finalizers and counter updates check the current target and resync ID, ignoring late results from removed, replaced, or canceled runs. Site state is actually written back, and a late Completed result cannot overwrite Canceled.
Success and failure require different finalization order
The result-consumer contract from #136 must remain intact: persist terminal state only after the consumer finishes.
Canceling before joining workers on a normal completion path could discard pending work and turn a successful run into Failed. The final code calls finish exactly once from one defer, using defer order to complete cleanup. It therefore needs no additional sync.Once.
WithCancelCause distinguishes an explicit user cancellation from parent-context interruption. User cancellation becomes Canceled; parent interruption observed during finalization must not leave an interrupted run marked Completed. Shutdown while a run is still waiting for a slot preserves the existing Pending state for restart recovery.
Protect terminal state and resumed work
Cancellation can still arrive between the context check and the terminal save. Under its lock, markStatus must persist an existing Canceled state even if the finalizer previously computed Completed. An old resync ID must also be unable to overwrite a new run.
This is not a transaction across metadata files. If a bucket completed and persisted before cancellation, a subsequent site cancellation can leave different records reading “bucket Completed, site Canceled.” Completion happened first. The guarantee is that late completion cannot turn an already canceled run back into Completed, not that cancellation erases work completed before it.
The recovery loader, loadResync, previously launched goroutines and then immediately executed defer cancel(). Inspection of SILO’s actual shared-lock implementation confirmed that this cancel ends the merged leader context; it is not a no-op. A WaitGroup now retains that context until resumed runs exit. Losing leadership still cancels the existing context; replacing it with a global context would bypass the leadership constraint. Loading disk state must also preserve newer in-memory start/cancel state.
Fable review and the complexity decisions
The review used the installed Claude Code with model argument claude-fable-5-1[1m] and --effort max. Baseline reproductions and the minimal proposal produced agreement on all three repairs. The final patch and validation were then reviewed again, yielding GO. The conclusion rests on code and evidence, not model agreement alone.
| Proposal or review point | Final decision |
|---|---|
| Treat a purge’s 405 probe as completion | Rejected: marker existence does not prove permanent deletion |
| Add bounded retries, backoff, and a persistent MRF scheduler | Not introduced: the existing queue and scanner already provide recovery; the demonstrated gap is visibility |
| Send more shared cancellation tokens | Rejected: this still cannot guarantee identity routing, broadcast, or isolation from later tasks |
| Add a separate cancellation tombstone registry | Unnecessary: existing target status and resync ID under one lock close the registration race |
| Give each run an owned context and cancelable blocking operations | Retained: full-queue deadlock and walker leaks have deterministic reproductions |
Protect finish with sync.Once |
Initial review required protection against double close; the final implementation has one deferred call site, and re-review accepted omitting Once |
| Wait for resumed runs before releasing leader context | Initial review required checking necessity; SILO’s cancel is effective, so the WaitGroup stays, with leadership-loss coverage |
The final review accepted two further implementation boundaries. The infrequent resync start operation holds the status lock across configuration reads and writes, consistent with existing terminal saves. The active registry uses resyncOpts, including resyncBefore, as its key; current callers reuse the same in-memory values, and no identity mismatch was reproduced. Future changes that reconstruct time values or restore task identity should revisit equivalence, rather than adding another registry without evidence now.
Validation and reproducible evidence
Regressions were added against unchanged production code first, and failed before the fixes. New tests exercise the production handler, real erasure storage, actual metric registration, and resyncBucket, rather than only testing helper logic that repeats the implementation.
| Validation area | Result and evidence |
|---|---|
| Single-delete purge | ErasureSD and Erasure source/target cleanup; legacy PENDING recovery; marker idempotency; real 403/405/503 failure semantics |
| MRF | Actual capacity-one overflow and retry exhaustion; administration JSON; registered v2/v3 counter types and values; object/delete worker affinity |
| Resync | testing/synctest coverage for blocked Walk, full worker queues, user and queued cancellation, unrelated IDs, later tasks, terminal races, stale IDs, slot release, and leader recovery |
| Complete local suite | go test ./... -count=1 -timeout=30m passed, with 50 tested packages |
| Concurrency and repetition | Focused replication race tests passed; cancellation regressions passed 100 repetitions |
| Tooling and contracts | Local build, vet, lint, generated-file checks, rebrand compatibility guard, and git diff --check passed; dependencies and compatibility baseline unchanged |
| Two native servers | Built from local source; direct single-object DELETE removed the target marker and source xl.meta entry while retaining the original data version; no downloaded Server Docker image |
| Remote integration | All eight PR checks passed, followed by all six main Go CI jobs and VulnCheck |
Tests pinned to the merged revision: delete markers, MRF visibility, and cancellation lifecycle. With that revision and Go 1.27.1, the relevant checks can be repeated with:
After complete local validation, only new-test formatting and fixtures changed: an existing ARN was reused, and the collector supplied the metric prefix, preventing the compatibility scanner from treating test strings as new protocol identifiers. The guard was not weakened. Relevant tests, lint, and compatibility checks were rerun after those adjustments, and remote CI checked the final commit.
| Evidence point | Exact identity |
|---|---|
| Baseline | 450dcb8484bc1337deba0cf608cc893a6691d794 |
| Final PR head | 66fe61ff65c83d68b74baa637a11623015c7aa21 |
| Merged main | d1105bbb3d4a0afa33b3a4ac11b821235038ed0e, with the same source tree as the final PR head |
| PR Go CI | 34320440012 |
| Main Go CI | 34321319278 |
| Main VulnCheck | 34321319274 |
Maintenance and release decisions
Future changes must continue to establish operation identity, visible failure, truthful per-object outcomes, complete terminal counters, and cancellation that releases its own resources. Neither an API returning Completed nor an object existing at the destination can replace those checks.
The code verdict is GO, and the delivery facts are main integration and passing CI. A formal release still requires selecting a tag, verifying packages and images, and establishing that deployments contain the repair. The scanner-dependent MRF recovery delay, the open cross-pool issue, and the externally reported 405 storm not reproduced on current SILO remain part of this decision record.
Second round (2026-09-16): worker-side purge classification and persisted MRF recovery
The first round classified deletes at the handler entry (#153), exposed MRF queue drops (#152), and completed resync cancellation (#137). The second round — PR #196, fix 0c61128d2, integration verification aea3882c9 — repairs a different surface: the replication worker’s own exits, the outer aggregation, and the persisted MRF recovery path for delete markers. The two rounds are complementary; neither subsumes the other.
Status: on verified main
40220bd836cb, not in Server 20260903. All evidence is synthetic (real single-drive and 16-drive storage, signed DELETEs, controlled HTTP targets, real persisted-MRF disk files replayed through a fresh worker pool). The externally reported 405 storm is not reproduced and not attributed to these paths.
What was still broken
- Legacy-shape tasks were skipped entirely. A task with an empty
VersionID, a non-emptyDeleteMarkerVersionID, a COMPLETED creation target, and a PENDING purge target never issued its DELETE — the per-target creation early-return suppressed it. - Fixing only the target function made aggregation lie. The outer status selection keys on
VersionIDand creation state, so a failed legacy-shape purge aggregated as COMPLETED, emittedObjectReplicationComplete, and skipped queueing to persisted MRF — worse than the baseline. - Persisted MRF dropped every marker 405. Replaying a disk MRF entry for a delete-marker version fetched real marker metadata plus
MethodNotAllowed, and the error path discarded it — the marker MRF recovery route was a dead end. - Failed purges overwrote successful resync markers, completed purges were re-sent, and a not-yet-ready HEAD unconditionally overwrote creation state.
- A multi-target empty-state regex misparse could corrupt creation state once. A disk marker carrying only creation metadata, replayed against a task with purge state, let two empty target states parse as a bogus
Pending; the deleted-flag guard then rewrote the whole creation block as empty entries with fresh timestamps — a one-shot corruption that requires disk/task divergence to reach.
Root causes: the worker had no task-level purge classification (the per-target predicate in the community’s #184 was itself wrong — a composite purge status can never reach COMPLETE through it; its investigation and proposed fix nevertheless shaped this follow-up, with thanks to Julien Laurenceau); MRF recovery had no valid-405 identity gate; and the delete task carried no retry count, so the existing mrfRetryLimit drop was unreachable on the delete path.
The repair
- One classification, every exit.
isVersionPurge()(non-emptyVersionID, or a non-emptyDeleteMarkerVersionIDwith a composite purge status) drives both the inner target function and the outer aggregation. Purge exits write onlyVersionPurgeStatusand leaveReplicationStatusempty — the storage layer’s “do not update” signal. - A three-field clear guard empties the composite status on the purge path, making the multi-target misparse unreachable at disk writes.
- Purges send the canonical permanent-delete request — explicit
versionId,ReplicationDeleteMarker=false, no HEAD/readiness probe (authorization is the DELETE’s own). This also prevents a lost-response retry from re-creating a marker at an unversioned fallback. - A valid-405 gate for MRF recovery. A
MethodNotAllowedschedules recovery only when the returned object is a delete marker, the bucket/object/version identity matches, and the modification time is non-zero. - A bounded retry budget. Delete tasks carry a retry counter, incremented at all three persisted-MRF entry points (aggregation failure, lock failure, queue-full fallback), respecting the existing limit; after exhaustion, the scanner can still re-raise healing.
- Audit and event status map
COMPLETEtoCOMPLETEDat the statistics/event boundary only, reusing the existing legacy constant.
What 405 means, precisely
For creation (replicating a delete marker), a HEAD 405 on the target marker version means already created — idempotent completion. For purge (permanently deleting a version), a 405 from MRF identity probing with full marker identity means work remains; the purge’s own success is decided by the DELETE alone, and a DELETE 403/405/503 is always a real failure. Empty or null version markers return ObjectNotFound, not 405, and sit outside the gate.
Boundaries that remain
The legacy in-memory task shape does not serialize across restart and has no current producer — its handling is robustness, not an active repair. Targets without a configured client still only log. Target-level resync replacing a purge subset is a pre-existing defect this round neither caused nor fixed. Persistence was driven directly in tests; timer-based flush and process-crash durability are not claimed. Multi-process site-replication meshes and cross-region acceptance are out of scope. For the tag-ordering and replica-metadata repairs in the same reliability series, see Replicated Tag Ordering and Replica Metadata Normalization.
4.9 - Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection
This page records the analysis and repair of two defects in how object tags
keep their ordering across replication, merged into Server main as
PR #193 (fix
03027727d) and
PR #196 (fix
680eac66e).
As of 2026-09-16: both fixes are on verified main
40220bd836cb. They are not in the published Server 20260903; use the linked PRs to identify a build containing them.
Delivery boundary: source acceptance (regression tests plus the R4–R8 integration run whose PR #196 checks passed) only. No tag, package, image, or production rollout is established by this record.
Evidence class: synthetic signed HTTP tests use real single-drive, 16-drive and multi-pool erasure backends; sender, wire-shape and precondition checks also include function-level tests. No customer incident is attributed to these paths.
The shared model: a tag value and its revision are one state
Tags replicate with an internal revision timestamp
(wire header X-Minio-Source-Tagging-Timestamp, stored as x-minio-internal-tagging-timestamp). The ordering rule on the receiving
side is simple: a replicated tagging state wins only when its revision is
newer than the stored one. Both defects in this family break that rule by
making the revision — not the value — the part that gets lost:
- R4 dropped the timestamp on the way into an SSE-KMS destination, so newer tag updates lost to stale stored state inside the storage-layer reconciliation.
- R5 meant an empty value (a deletion) carried no revision at all, so the protocol could not express “deleted at time T” — and a delayed event could resurrect what a client had already deleted.
R4: SSE-KMS destination copies dropped tag revision timestamps
Failure form. A trusted replication COPY to an SSE-KMS encrypted
destination returned 200, the object was correctly encrypted, plaintext GETs
worked — and the destination’s tags and their timestamps stayed at the old
values. Because the HTTP request succeeded, nothing surfaced the loss. The
storage-layer reconciliation (reconcileStoredObjectTags) compares revisions
under the object write lock; without the incoming timestamp, the old stored
state won.
Trigger surface. Not only explicit SSE-KMS headers. Bucket-default KMS and global automatic encryption hit the same code path, so the defect could fire with no KMS header in the request at all.
Root cause. The option builder for PUT-like requests parsed the trusted
source-tagging timestamp into ObjectOptions, then the SSE-KMS branch
constructed and returned a different ObjectOptions carrying mtime, ETag,
replication trust, and two Object Lock timestamps — but not
ReplicationSourceTaggingTimestamp. The omission dated back to upstream
c4373ef290 (2021); a 2026 Object Lock repair added two more timestamps to
that literal and still missed this one. Before R5, the consuming path was COPY ordering. R5 adds timestamp persistence for replica PUT and multipart initiation, as well as the duplicate-precondition consumer. Those SSE-KMS paths also depend on R4 preserving the option, so backports must consider the pair together.
Fix. One field added to the existing SSE-KMS ObjectOptions literal
(03027727d), nothing
else. Regression tests cover every destination encryption (none, SSE-S3,
SSE-KMS with and without key context, SSE-C) crossed with trusted/untrusted
source, missing/valid/malformed timestamps, and 50 signed ordered COPY+GET
cycles across two single-pool backends (single-drive and 16-drive erasure)
with 1–3 ns event spacing. The KMS cases use a test stub.
No backfill. A lost source-tagging timestamp cannot be reconstructed at the destination. After upgrading, new tag events replicate in order; replay of old events still follows the timestamp comparison: an incoming event must be strictly newer; the stored value wins ties and rejects older events.
Deferred observation. The same SSE-KMS literal also omits the proxy and speedtest option fields; the speedtest flag is read on the storage path, so global auto-encryption would drop it on a speedtest PUT. Recorded here as a separate follow-up, without asserting a public issue exists; deliberately not bundled into this repair.
R5: empty tag values had no revision, so deletions could resurrect
Failure form. Nine baseline regressions across real-storage and function-level tests:
- A successful
DeleteObjectTaggingnever minted a new revision, so a later-arriving trusted metadata COPY with an older view of the tags re-instated them. - A newer COPY carrying an empty tagging state was ignored — empty meant “nothing to say” instead of “deleted”.
- The first replica PUT parsed the source tag timestamp and never persisted it.
- Equal visible tag values collapsed to “no replication needed”. HEAD does not expose a tag revision, so a newer deletion or re-addition was invisible and ordering was not restored.
- A queued replication event’s completion callback wrote the old tags from its snapshot back over an already-committed deletion — and without a timestamp, the resurrected set inherited the deletion’s newer revision, which is worse than the originally-reported symptom.
Root cause. A tag value and its timestamp (including the empty value’s timestamp) constitute one state. The old protocol could only represent non-empty states: DELETE-tagging never minted a revision, the sender only attached timestamps in the non-empty branch, and the receiver only made decisions in the non-empty branch.
Fix (680eac66e):
- Every tagging mutation mints one revision. PUT/DELETE tagging handlers unconditionally stamp a single UTC RFC3339Nano revision, whether or not replication selects the object. The storage layer enforces monotonicity under the write lock: a local revision that is not strictly newer is advanced to stored+1 ns, and multi-pool backends compute one value that strictly exceeds every pool’s copy.
- The sender transmits tombstones. Empty values carry their recorded revision; empty-without-revision is not fabricated; a malformed recorded timestamp fails closed rather than being silently repaired.
- The receiver accepts tombstones. Replication COPY captures the stored tag pair before rebuilding, so an empty value with a timestamp enters the existing reconciliation as a state that can win. Duplicate suppression is relaxed only for a strictly newer source revision. Multipart completion orders tags from the persisted upload metadata. The delete-acknowledgement path no longer writes snapshot tags back. Ordinary SSE-C rotation drops the old tag timestamp from copied encryption metadata so it cannot overwrite the new local revision.
Operator-visible changes:
- Equal timestamps now resolve to stored-wins on unqualified and explicit null COPY requests. The old handler let the incoming state win ties on the unqualified path; the change is a compatibility-visible tightening in the correct direction.
- An ordinary COPY with
tagging REPLACEand no tags now genuinely clears destination tags. The old default-metadata path carried source tags over — an S3 consistency improvement, but a behavior change. - Every ordinary COPY (including key rotations that change nothing) records one new local revision.
- Both ends of a replication pair must upgrade together. An old peer still drops empty-value revisions, so a new sender’s tombstones are invisible to it.
- Objects with recorded revisions incur one extra metadata COPY per object during explicit resync/heal. When the destination has bucket-default or automatic KMS encryption, that “metadata” COPY rewrites the object data — budget accordingly for bulk resync.
A nonempty legacy tag set without a revision uses object ModTime as its sender fallback; an empty set without a revision does not acquire a fabricated tombstone. A strictly newer trusted tag revision bypasses only the internal ETag/version duplicate guard (otherwise 412 PreconditionFailed); client If-Match and If-None-Match remain enforced.
The tag repair adds no wire field or storage format and has no capability negotiation. Both endpoints are needed for ordered deletions; an old hop retains the old behavior. This statement does not authorize rolling upgrade or downgrade of the entire September candidate, which also contains the separate IAM migration. Reconcile already-damaged tags from an authoritative source by a new explicit tag change; the lost historical order cannot be reconstructed.
Limits, stated as limits:
- No migration: history with missing or wrong deletion timestamps cannot be reconstructed, and no historical tombstones are fabricated.
- Replication rules with tag filters evaluate target eligibility against the post-deletion (empty) tagging state, so a rule filtered on the deleted tag never sees the deletion. This is a pre-existing scoping decision, unchanged here.
- Arbitrary clock skew is not a total order. The repair establishes per-hop ordering, not multi-site causality.
- A malformed recorded timestamp makes the sender’s construction fail permanently and the event retry through MRF until an explicit, correct tag change replaces it — deliberately fail-closed.
Rejected alternatives
- Synthesize a tombstone from ModTime for empty-without-revision objects. Every object that never carried tags would gain a revision; combined with forced metadata replication, every object would take the metadata-COPY path on every hop.
- Transmit only when the value is empty. Withdrawn by its own proposer
during review: the equal-value re-add sequence (set X at T1, delete at T2,
re-add X at T3) loses the re-add’s revision under that condition. A
regression test (
TestTaggingRepeatedValueNeedsRevisionDelivery) pins the counterexample. - A new HEAD revision protocol. The pinned
minio-gometadata extractor discards internal response headers, so this needs a new wire contract for marginal benefit; the worst case is still a forced metadata COPY. - A distributed causal clock, or regenerating timestamps at commit. The wall-clock model plus the in-lock monotonic guard is the minimal correct fix.
- Per-pool monotonic guards only. Ordinary reads return single-pool object info, so a sender could emit a stale primary-pool revision; the unified multi-pool value is required.
Verification and what it does not prove
R4 regressions cover the encryption × trust × timestamp matrix and the ordered sequence on two single-pool backends (single-drive and 16-drive erasure), with a stub KMS. R5 separately has a multi-pool tagging-deletion regression. The R4–R8 integration run repeated targeted tests on the merged tree, including the isolated R5 multi-pool test; all eleven PR #196 checks passed. These establish the tested ordering cases. They do not establish multi-site scheduling under real clock skew, cross-region failover, or behavior of deployments that upgrade one end of a pair only.
The upgrade summary and release boundary live in the component matrix; the sibling repair that stops normalized replica metadata from being re-injected is recorded separately in Replica Metadata Normalization.
4.10 - SILO Server 20260903 Pre-release Review
This is the durable pre-release engineering record behind SILO 20260903. It explains why an earlier “all issues are solved” assessment was not accepted at face value, what the independent review found, how the fixes were narrowed, and which gates still remained at the review point. The linked release note records the later publication result.
Decision: the source candidate at
6e112d1856d4f3655f30fc81ee47e9f43d50d8f3is a code-level GO for remote review. Production release remains a conditional GO until remote CI, Test Release, tag and artifact verification, signing, container publication, and public pull checks complete.
Baseline:RELEASE.2026-08-06T00-00-00Zat3be10fcc1a44f6620ded0bd303461f9d688cca23.
Scope: SILO Server behavior and its embedded/pinned runtime components. Documentation, the standalone Console, mcli, package repositories, images, and the deployed site are separate deliverables.
Publication closure: the later final tree9b11dc9469e650815b775cb47b039610644f5da4was published asRELEASE.2026-09-03T13-18-01Zon 2026-09-04 after the remote, package, provenance, container, and public-download gates below completed. The conditional decision in this page remains the historical review criterion, not the current release state.
2026-09-09 follow-up: the repairs and main validation for #153, #152, and #137 are recorded separately in Replication Reliability. That record preserves #136’s accounting contract, explains cancellation lifecycle decisions, and keeps #133 separately tracked. It does not change the historical assessment of the 0903 release candidate below.
Why the second review was necessary
The first implementation pass had strong test results and resolved most reported defects. Its conclusion was nevertheless too broad: it treated green tests and a clean worktree as proof that every security invariant had been closed.
An adversarial review asked different questions:
- Can the same invariant be bypassed by a different valid wire representation?
- Does a pre-authentication fast path still perform I/O or acquire state?
- What happens when metadata exists but cannot be loaded?
- Do two individually correct read-modify-write paths share the same serialization boundary?
- Does request sanitization preserve all SigV4 streaming state?
- Does a validation claim describe the final tree or an earlier one?
- Is a complex mechanism protecting a reproduced failure, or only a hypothetical future?
That pass found real defects after the initial “ready” claim. The correct response was not to distrust all prior work, but to narrow every assertion to an invariant and an observed tree.
Review result by area
| Area | Adversarial finding | Final resolution | Status |
|---|---|---|---|
| Bucket metadata | Independent config locks could lose updates to the shared .metadata.bin record (#102) |
One bounded metadata.lock surrounds every whole-record writer, migration, import, adoption, and healing path; changed-field replication avoids stale whole-record replacement |
Closed in candidate |
| Bucket creation | ForceCreate and site adoption could replace existing config with defaults |
Preserve existing records and update only creation/adoption state; add regression tests for clobbering | Closed in candidate |
| Object Lock | Comparing lock-document bytes with one canonical XML document missed valid configurations carrying a Default Retention rule | Parse Object Lock first, then derive the versioning invariant from the parsed enabled state; verify update, read-back, and disk reload | Closed in candidate |
| Pre-auth CORS | Arbitrary path segments could cause metadata reads and cache growth | CORS lookup reads resident metadata only and does no object-layer I/O | Closed in candidate |
| CORS startup | A nonresident name could fall back to global CORS before metadata initialization | Preserve an explicit fail-closed startup state | Closed in candidate |
| CORS load failure | Forgetting that a real bucket failed to load made it indistinguishable from a nonexistent bucket and exposed the global fallback to pre-signed requests | Maintain a bounded failed-bucket set, clear it on every successful load/remove/refresh path, and keep those buckets fail-closed | Closed in candidate |
| CORS recovery | A successful on-demand GetConfig reload did not initially clear the load-failure bit |
One-line final fix 84e1580a4 plus targeted race coverage |
Closed in candidate |
| Replication trust | Presence of client-controlled internal headers enabled privileged behavior in multiple handlers | Authenticate first; require an exact marker plus s3:ReplicateObject or s3:ReplicateDelete; carry a private context decision; sanitize untrusted headers afterward |
Closed in candidate |
| Streaming uploads | The sanitized request clone did not initially share the original trailer map | Preserve the trailer map so late-arriving streaming checksums remain visible | Closed in candidate |
| Snowball | A request-wide trust bit could leak between extracted entries | Derive and isolate trust per entry; preserve request defaults across workers | Closed in candidate |
| SSE-C | Zero-byte reads and GetObjectAttributes could skip customer-key authentication |
Require a successfully unsealed key, with a separate authorized-replica exception | Closed in candidate |
| Delete authorization | Explicit version deletes checked the ordinary delete action instead of requiring s3:DeleteObjectVersion |
Align single and multi-delete authorization, keep replication deletes on s3:ReplicateDelete, and preserve auth/audit context |
Closed in candidate |
| Admin authorization | User/group status changes always checked the enable action | Check the action that matches the target state | Closed in candidate |
| Checksums | Multipart and copy paths omitted fields, accepted invalid combinations, or computed over the wrong representation | Complete algorithm/type validation, server-side part calculation, federated propagation, AWS errors, and CopyObject transform ordering | Closed in candidate |
| Release evidence | Full acceptance initially described a tree that changed afterward | Record full acceptance at ebac0ca73 and current-tree targeted gates separately |
Closed as an evidence defect |
The invariants that now define the candidate
Trust is derived once, after authentication
An internal-looking header is still client input. The request must first pass the existing authentication path in its original signed form. Only then can the handler combine:
- an exact, single replication marker;
- a non-anonymous authenticated identity;
s3:ReplicateObjectors3:ReplicateDeleteon the addressed resource;- replica status where the narrower replica-only semantics require it.
The result lives in private request context. Header stripping is defense in depth for legacy consumers, not the source of authority.
This ordering matters because SigV4 may sign the headers. Sanitizing first would reject legitimate replication with SignatureDoesNotMatch. The sanitized clone also has to share the request trailer: trailers arrive after the initial header parse and carry streaming checksums.
The complete receiver-wide model is in No I/O Before Auth, No Privilege From Headers.
A shared record has one write boundary
Policy, lifecycle, SSE, tags, quota, replication, Object Lock, versioning, and CORS are logical fields but physical members of one bucket record. A per-field mutex cannot protect a whole-record read-modify-write.
The selected repair is deliberately smaller than a new database or transaction layer:
The lock does not cover object data I/O and is bounded to a bucket-metadata operation. Migration and healing must participate because they also replace the whole record. Replication receivers merge only changed fields so an older remote snapshot cannot erase unrelated local state.
Failure is a state, not the same thing as absence
The CORS hot path must distinguish four states:
| State | Result |
|---|---|
| Metadata system not initialized | No CORS headers |
| Known real bucket whose metadata load failed | No CORS headers |
| Resident bucket with a bucket CORS document | Evaluate that document |
| No resident metadata and no known failure | Use the server-wide fallback |
The second row is why a failed-bucket set survives the simplification pass. A pre-signed URL is already authorized by its signature and may access a private object without bucket-policy evaluation. In that case the bucket CORS document is the browser-origin boundary. Losing the failure bit and using a permissive global fallback would weaken that boundary.
The set remains bounded by real bucket load attempts and is maintained through two helpers. Successful load, removal, stale-bucket cleanup, refresh, reset, and concurrent load all have tests.
Object Lock is semantic, not textual
Any valid enabled Object Lock configuration implies versioning. XML whitespace, element order, and the presence of a Default Retention rule do not change that meaning. Therefore normalization follows parsing, not a byte comparison against one canonical document.
The resulting versioning record is plain Enabled. A suspended state and an exclude-prefix extension are incompatible with the lock invariant and are removed on update, read-back, and reload.
Complexity audit
The pre-release pass explicitly looked for over-design, duplication, defensive programming without a threat model, and stale compatibility machinery.
Complexity retained because it protects a reproduced failure
- One metadata lock: retained because a deterministic cross-type lost-update test reproduced data loss.
- CORS tombstones: retained because site replication cannot distinguish deletion from “never observed” without them.
- CORS load-failure state: retained because a pre-signed URL provides an authenticated, policy-independent counterexample.
- Two replication trust levels: retained because ordinary replication and replica-ciphertext/SSE semantics do not use identical wire shapes.
- Post-authentication sanitization: retained because sanitizing before SigV4 verification breaks legitimate signed requests.
- Adversarial multi-pool/null-version tests: retained because single-pool happy paths do not exercise the state-selection failures they caught.
Complexity removed or narrowed
- CORS failure-set mutations were centralized in
noteLoadFailureandclearLoadFailure. - The replication import path now applies changed fields rather than copying an entire possibly stale record.
- Obsolete encryption helpers, dead event-target functions, and abandoned handler branches were deleted.
- The compatibility guard stopped inventorying every exported source symbol and now protects the actual served routes and frozen wire/configuration surfaces.
- The old
wait_pipelint exemption was removed;gomodguard_v2replaced deprecated configuration. - Dynamic timeout tests no longer call global
rand.Seedfrom a parallel package. - The server returned from the temporary
silo-gofork to the reviewed upstream-compatibleminio-gorevision.
Changes deliberately not introduced
- no generic metadata transaction framework;
- no second CORS cache or unbounded negative cache;
- no new public “trusted replication” request header;
- no cross-repository release gate that makes the server depend on a later Console or documentation release;
- no partial conditional-delete contract in the release candidate;
- no broad rewrite of inherited site-replication registers without dedicated convergence tests.
Deferrals and why they do not all have the same severity
| Item | Classification | Release decision |
|---|---|---|
| Conditional delete #10 | Inherited missing S3 feature; dangerous only to callers that assume unsupported If-Match / per-object ETag is enforced |
Document prominently; do not merge the incomplete PR or a single-only half contract |
| Multi-site config deletion #77 | Inherited convergence defect for policy/SSE/tags/quota; CORS has its own fixed register | Not a single-site blocker; deployment condition for users relying on those multi-site deletes |
ListMultipartUploads #79 |
Inherited listing-conformance gap | Known issue; not a data-integrity blocker for ordinary multipart workflows |
Federated CopyObject #99, #100 |
Legacy-backend checksum/inline-object gaps | Block use of the affected features, not the general server release |
| ILM relocation PR #60 and broad SSE issue #61 | New capability requests | Outside the release safety boundary |
“Inherited” does not mean harmless. It means the defect was not introduced by this change set and should be evaluated against the documented release contract. A deployment that depends on one of the affected paths inherits a deployment-specific stop condition even when the general release remains conditional GO.
Evidence
Full acceptance tree
The full local acceptance corresponds to ebac0ca73bbf251b070bb6df4d8005015841f901:
- full
cmdandinternalsuites; - complete
cmdrace suite: 365.448 seconds, pass; - lint: 0 issues;
- rebrand/compatibility and generated-file guards;
govulncheckwith no reachable vulnerability;- six
make verifydeployment shapes: 174 PASS / 0 FAIL.
The first two make verify attempts encountered environment/setup failures while obtaining mcli, not test failures. The successful run used the locally checksum-pinned mcli, retained the outbound proxy for GitHub downloads, bypassed it for localhost, and placed GNU userland tools first in PATH. That distinction is part of the evidence rather than something to hide.
Post-acceptance candidate
The only code change after that full run is 84e1580a4, which clears one CORS failure-state bit after a successful on-demand metadata reload. The candidate merge adds no code; 6e112d185 changes only Helm release metadata and documentation. On the final candidate, the following pass:
git diff --check;- targeted CORS and Object Lock
go test -race; - rebrand guard;
- generated-file check;
- lint with 0 issues.
- Helm lint, default and optional renders, chart packaging, and the seven-resource legacy-upgrade identity guard.
This evidence is proportional to a one-line state-transition fix, but the remote CI and release workflows must still run against the pushed tree.
Go, no-go, and ownership of the remaining gates
Code decision: GO
No confirmed code defect from the two review rounds remains unresolved in the candidate. The fixes are covered at the layer where their invariants live, and the retained complexity corresponds to reproduced counterexamples.
Production decision: conditional GO
The server must not be described as released until all of these are facts:
- candidate commits are pushed and reviewed;
- remote CI and Test Release pass on the pushed head;
- the intended tag points at the reviewed chart 7.0.2/server 0903/client 0903 release tree;
- Draft artifacts, checksums, SBOMs, attestations, and signed RPMs verify;
- finalize and Docker release publish both classic and distroless variants;
- anonymous download and pull tests pass;
- release notes are updated from the tagged facts and the documentation site is deployed.
Any failure in steps 1–6 is a release blocker. A local green suite cannot substitute for them.
Deployment-specific stop conditions
Operators should delay even a successfully published release when they cannot yet:
- update every node in a distributed cluster within one coordinated maintenance operation;
- update every member of a site-replication group before using bucket CORS;
- revise IAM policies for
s3:DeleteObjectVersionand status-action separation; - avoid or explicitly accept the known #10, #77, #79, #99, or #100 path their workload depends on.
The final conclusion at the review point was intentionally narrower than “everything is fixed”: the reviewed candidate was ready to enter the release machinery, the remaining limitations were explicit, and production publication was gated by verifiable artifacts rather than confidence. Those gates later completed for the release linked above; these deployment conditions describe Server 20260903, the release this record reviewed.
Restart and readback verification (2026-09-11, #116)
A later acceptance (#116, run on 2026-09-11) measured bounded restart/readback behavior, including the inherited readiness gap. The durable part is the method, which any operator can reuse when validating a restart or upgrade window:
- Acknowledgement ledger. Every acknowledged PUT writes to a unique versioned key, and the acknowledgement records the VersionId, byte count, and SHA-256 immediately. Readback fetches by exact VersionId and asserts all three — an early confirmation cannot be silently replaced by a later write.
- Readback timing. Periodic re-reads happen at 15/30/60 s after the data canary succeeds, through every peer, and the final check includes writes made during a single-node outage and after its rejoin.
- No retry masking. Each canary carries a hard 60 s deadline covering setup, request, response-body read, and sleep, and SDK retries are disabled so a recovery window cannot be papered over by client-side retries.
- Driver topology. Four Linux/arm64 containers on one host with independent network identities, one drive each (EC 2+2), and tmpfs volumes kept mounted by a holder container through the full shutdown; nodes stop in parallel with a 10 s grace period, then start in parallel.
The operational finding worth remembering: admin-endpoint readiness is not data readiness. The fixed historical record tested 0806, 0903 and the then-current main repair build. The main build reached the admin gate 2.461 s after restart, then needed another 14.489 s for the data canary: different timing origins. The 0903 run took 0.191 s after its gate, which does not establish immunity to the inherited startup window. A readiness probe against admin/health says nothing about the data plane during that window, and no fixed sleep substitutes for an actual data-plane check.
Boundaries, stated as boundaries: this acceptance covers process/container restart and TCP peer reconnection on a single Linux host. It does not prove persistence across independent hosts, host reboots, or physical media failure, and the timings are individual observations, not latency guarantees. The run artifacts are retained outside the documentation tree; the method above is the part that generalizes.
September 16 source follow-up: #10 closed through the independent single-object repair #145; batch ETag deletion is still absent. #77 and #99/#100 were repaired on main. #133/#144 were subsequently addressed by #178; #79’s default listing limitations remain open. These changes must not be retroactively attributed to Server 20260903. See the current component/source matrix.
4.11 - Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject
This page records the repair of how a trusted replication receiver restores
metadata for a replica, merged into Server main as
PR #194 (fix
4fcdf37ce, merged as
9f3037e941).
As of 2026-09-16: the fix is on verified main
40220bd836cb. It is not in the published Server 20260903.
Provenance: the production logic follows PR #187 by Mikhail Khadarenka; the merged change keeps that authorship and narrows it to a review-validated boundary.
Evidence class: an HTTP-level baseline of 64 leaf cases (44 controls passing, 20 defect failures before the fix) against real single-drive and 16-drive erasure backends, plus counterfactual replay of the same suite against the baseline helper. No customer incident is attributed.
What went wrong
The ordinary PUT path normalizes metadata: it strips the transport-only
aws-chunked token from Content-Encoding, and removes the
X-Amz-Meta-X-Amz-Unencrypted-Content-Length/-Md5 user-metadata keys that a
GHSA-76wf-9vgp-pj7w mitigation deliberately deletes. The trusted replication receiver,
however, restored replica metadata by re-running the same permissive
extractor with replication allowed — replaying every supported header and all
user metadata from the original request. Concretely, on a trusted replica
write the server could store and later return via GET/HEAD:
Content-Encoding: aws-chunked(a pure transport encoding that must never be stored, per the AWS SigV4 streaming rules), or the unsplitaws-chunked,gzipstring instead ofgzip;- the two GHSA-redacted user-metadata keys — a partial rollback of that mitigation, limited to trusted replica writes;
- for Snowball entries without their own PAX header: the outer archive’s content-type, cache-control, and user metadata.
The object bytes themselves were not necessarily damaged; the stored metadata
was wrong. The regression was introduced by
56fa63bfd
(2026-04-15, the replication header trust boundary hardening, CVE-2026-34204)
— whose trust protection is correct and stays.
The fix
One file (cmd/handler-utils.go). The boolean dual-mode helper is deleted:
- the ordinary extractor unconditionally skips replication-only keys;
- a new replica extractor walks only the replication-to-internal header map and restores only the six replication-scoped fields: the SSE-C sealed key material, sealed algorithm, IV, and encrypted-multipart marker (the empty marker is honored by key presence), the actual object size, and the SSE-C checksum identity mapping;
- it never re-reads ordinary supported headers or user metadata.
Expected stored encodings after the fix:
| Requested encoding | Stored Content-Encoding |
|---|---|
aws-chunked |
none |
aws-chunked,gzip |
gzip |
gzip |
gzip |
aws-chunked, gzip (note the space) still stores gzip with a leading
space, and gzip, aws-chunked stores the whole string. These are
documented status quo, asserted by tests as such — not claims of repair.
Operator-visible changes
- Trusted Snowball entries no longer inherit the outer archive’s ordinary metadata. Without PAX records, this removes outer content-type, cache-control, expires and user metadata; with PAX records it removes fields the entry did not restate. Per-entry
minio.metadata.*still applies, the six replication-scoped fields still apply to authorized entries, and the archive’s storage class remains inherited. The in-tree batch producer callsPutObjectsSnowball, whose SDK emits the auto-extract marker, but it does not mark the outer request as a trusted replica; it did not use the affected inheritance path. - The GHSA-redacted keys are no longer written back on replica restore — matching what every ordinary PUT already did.
- Authentication, permission gating, and replication trust semantics are unchanged; the ordinary extraction path is byte-for-byte equivalent.
- Rolling back the code reopens the injection path but does not repair already-stored metadata.
Upgrade does not fix stored objects
An upgrade stops new pollution; it does not scan or rewrite existing objects. Two consequences matter:
- Objects whose authoritative source remains polluted can be selected repeatedly for metadata replication when comparison with a normalized replica detects a difference. Fix the authoritative source first, then let the copies converge.
- Ordinary S3 self-COPY is not a general remediation API: it can create new versions or shift timestamps rather than rewriting one version’s metadata in place.
The read-only audit runbook now provides an executable inventory tool and classification rules. It does not authorize or perform repairs.
The stored-metadata remediation proposal — status
A design for a future operation exists: build an inventory (including
non-current versions, not just the latest), verify by comparing against the
trusted source version or an independent checksum — never by guessing from
the wrong response header, and never by re-decompressing gzip just because a
label says so — process the authoritative source’s exact versions first,
then converge copies; keep immutable inventories and metadata backups; use
small validation batches with a rehearsed rollback. Where no supported path
exists for an object, stop and leave it — editing xl.meta directly is not a
supported operation.
Any selected procedure must preserve the required version identity and current version relationship, Object Lock retention/legal hold, tags, replication state, and encryption context. Check for concurrent changes before writing; blocked or unverifiable versions stay untouched. A local clone must prove the chosen operation and rollback before this proposal becomes an executable runbook.
This is a design proposal awaiting separate approval, not an executed procedure. No production inventory scan, object write, version change, or deployment has been performed as part of it. Treat it as the shape of a future runbook, not as a validated one.
Known limits
- The POST-form upload path (
bucket-handlers.go) calls the low-level extractor directly and never normalized encodings; that behavior is unchanged and flagged for a separate issue. - Local verification ran with a test-only capacity accommodation (a full host disk); the merged-tree rerun in the R4–R8 integration record covers the unchanged-tree case.
- No two-site scheduler, restart, or network-failure acceptance is claimed.
Verification
Regression tests (TestExtractReplicationMetadata*,
TestAPIReplicaContentEncoding, TestAPISnowballReplicaContentEncoding,
plus race-included trust/SSE-C round-trips) cover the mapping table, the six
restored fields, and the ordinary path’s equivalence; the recorded counterfactual run against the baseline helper produced 36 expected failures (20 HTTP cases and 16 helper cases), with 44 controls passing. These are the original repair’s observations, not a new execution by this documentation update. The upgrade summary lives in the
component matrix; the
sibling tag-ordering repair is recorded in Replicated Tag
Ordering.
4.12 - Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts
This page records the repair of Server’s HTTP read deadlines, merged into
main as part of PR #196 (fix
055030ea5).
As of 2026-09-16: the fix is on verified main
40220bd836cb. It is not in the published Server 20260903.
Evidence class: synthetic — a direct TCP comparison (header limited to 100 ms, header arriving over 400 ms, standard Go refuses where old SILO returned 204), plus a real single-drive process probe where flag and environment settings both rejected a 400 ms slow header while healthy requests continued. No production incident is attributed; the issue that motivated the investigation is a slow-HTTP DoS scanner report that has not been reproduced against a deployed cluster.
Two timeout classes, one connection
- Request-header absolute deadline.
ReadHeaderTimeoutbounds the total time from the start of header reading to its completion. Trickle-feeding bytes cannot extend it. On HTTP/1 keep-alive connections, Go first usesIdleTimeoutwhile waiting for the next request’s initial bytes, then starts a fresh header deadline.ReadHeaderTimeoutalso participates in the separate TLS-handshake timeout calculation. - Body rolling idle timeout. Once headers parse and the connection enters the active phase, the existing rolling semantics return: every successful read extends the deadline, and only a stall between bytes (the configured idle timeout) kills the connection. A long upload or download that keeps making progress is not capped in total duration by this HTTP/1 repair; other protocol, proxy, and application timeouts still apply.
Before this repair, the first class did not exist in practice: the
connection-layer wrapper replaced the socket deadline with
now + idle + 250 ms before every partial read, overwriting whatever
absolute deadline net/http had set — so a slow reader could hold a
connection open indefinitely by sending one byte per idle window.
Two independent defects
- The connection layer neutralized the absolute deadline. The
DeadlineConnwrapper’s read path reset the socket deadline on every partial read, defeating the read-header deadline Go’s server sets. The direct-TCP baseline proved it in isolation: with a 100 ms header limit and a 2 s idle window, a header that finishes at 400 ms was accepted. - The configuration never reached the server. The CLI accepted
--read-header-timeoutandMINIO_READ_HEADER_TIMEOUT, parsed defaults and all — and the server-context builder copiedIdleTimeoutwhile droppingReadHeaderTimeoutentirely, so the running server always saw zero. Fixing defect 1 alone left the real process accepting slow headers; the second fix is one line next to the idle-timeout binding.
The reason this stayed invisible for so long: the flag’s default (30 s) equals the idle timeout’s default, and with the flag unwired the server fell back to exactly that same 30 s — so every observable default behaved as if configured.
Configuration
- Flag:
--read-header-timeout(Hidden: true, absent from ordinary CLI help) - Environment:
MINIO_READ_HEADER_TIMEOUT - Default: 30 s (equal to the idle timeout default)
- There is no YAML configuration field for either timeout; the value binds once at startup from flag > environment > default.
| Setting | Effect |
|---|---|
| header > 0 | Absolute cap on HTTP/1 header phases; participates in Go’s TLS-handshake read window (including the HTTP/2 handshake) |
| header = 0 (explicit) | Falls back to Go’s rule: the read timeout (= idle timeout) applies; the CLI default is 30 s |
| header < 0 | Disables the header-specific cap. Positive read/write timeouts still bound TLS handshake reads, and positive IdleTimeout still bounds the keep-alive wait. This does not disable every connection timeout. |
| idle shortened, header unset | Header phase independently uses the 30 s default — the one combination looser than a naive expectation, though still strictly tighter than the pre-fix unbounded extension |
A negative value reopens unbounded slow-header trickling; it is not a recommended compatibility setting. An incomplete header cut off by the deadline normally sees a closed connection, not a guaranteed HTTP error status. The trigger was @AEGEGE’s scanner report #183; PR #195 was integrated through #196. The experiment does not establish reproduction in that deployment.
What each protocol gets
- HTTP/1: headers and keep-alive waits are absolute; the body keeps the rolling idle timeout. The connection-state hook composes with (rather than replaces) any caller hook.
- TLS: handshake reads take the minimum of the positive header deadline and the existing read/write timeouts; after the handshake, a fresh header limit begins. The handshake’s write side remains rolling — this repair is not a complete TLS-handshake resource limit.
- HTTP/2: untouched. When h2 is negotiated, the phase switching is
skipped entirely; h2 keeps its own native per-stream read timeout, which is
absolute, and
ReadHeaderTimeoutnever enters the h2 configuration. - Internal callers: Linux internode dialing uses its own rolling semantics; grid-hijacked connections unwrap to the raw TCP connection before any of this applies.
Rejected alternatives
- Clamp all future deadlines globally. Go 1.27 sets a whole-request deadline in some paths; with the read timeout equal to the idle timeout, this would hard-cap entire HTTP/1 requests — header plus body — and kill every large upload.
- Drop the read timeout and reinterpret zero as rolling idle. Zero is
net/http’s “never time out” for background reads and hijacked connections; reinterpreting it would break long handlers, and h2 would lose its per-stream timeout. - Wrap the body reader / response controller. Full chunked/drain/EOF accounting with h2 special cases is a far larger change than the header defect requires. (A later, unmerged branch explores a body-side response controller for the same DoS family; as of this record it is not part of main and not part of this repair’s claims.)
- Reconstruct the standard library’s deadline arithmetic in the hook. Duplicates stdlib internals that drift between Go versions; remembering the value stdlib actually asked for is the robust form.
- Auto-derive strictness from value comparisons. With all three defaults equal at 30 s, “shorter than the idle window” is indistinguishable in production defaults; such logic only works in test configurations.
Verification and limits
Tests pin the connection wrapper across three consecutive update periods (no extrapolation of the absolute cap), the phase transitions over HTTP/1 keep-alive, TLS, and HTTP/2-only negotiation, and the flag/env binding on the real CLI context; a process probe exercised a live server with a 100 ms header limit rejecting a header that takes 400 ms. Known limits: the TLS handshake write side stays rolling; handler CPU/storage waits have no deadline; the absolute header cap carries no slack while the rolling idle keeps its ~250 ms update slack; and multi-node, cross-region long-transfer acceptance is future work — the integration record explicitly does not count a scripted S3 long transfer as passed for this repair.
The upgrade note (a shorter header timeout also narrows the TLS handshake window; it is not a total-duration limit for uploads or downloads) is in the component matrix.
4.13 - Conditional DELETE: One Condition, One Logical Object
Status, 2026-09-16: PR #145 added single-object conditional deletion; PR #178 subsequently repaired multi-pool serialization and reconciliation. Neither change is in published Server 20260903. Check the component matrix before relying on this behavior.
The August proposal around PR #12 was broader than the code that merged. In particular, its proposed batch rejection, extra read authorization and current-version-only rule are not implemented guarantees. This page describes the maintained source at f99ed829b.
Implemented contract
| Request | Current main behavior |
|---|---|
Single DeleteObject, nonempty If-Match: <ETag> |
Check the client-visible ETag under the deletion lock; mismatch returns 412 before deletion |
If-Match: * |
Require an existing, non-delete-marker representation |
Explicit versionId |
Evaluate the addressed version, not an unrelated current version |
No If-Match, or an empty header value |
No conditional callback is installed; ordinary deletion applies |
Batch DeleteObjects with per-item <ETag> |
The request model has no ETag field; the XML field supplies no deletion protection |
Internal recursive x-minio-force-delete |
Prefix deletion returns before the object-condition path; do not use it as conditional deletion |
These conditions do not add an s3:GetObject authorization check. Ordinary deletion authorization still applies, including s3:DeleteObjectVersion for an explicitly addressed version, Object Lock checks and the separate trusted-replication path.
Why the condition belongs above individual pools
An object can have copies of different ages in more than one pool. A request condition concerns the logical object selected for the operation. Evaluating and mutating independently in each pool can delete one copy and then return 412 from another, or delete the latest copy and expose an older one.
The multi-pool path therefore acquires the shared namespace write lock, gathers the relevant states, evaluates the condition once, clears the callback for lower layers, and reconciles the selected deletion across pools. The same lock boundary must cover concurrent writes and metadata updates. This is the purpose of the later multi-pool repair, not a claim that every physical disk is atomically updated.
Counterexamples from the early design
With old ETag A in one pool and current ETag B in another:
If-Match: Amust not delete A first and only then fail against B.If-Match: Bmust not remove B while leaving A to become visible again.
A per-pool HTTP callback also risks concurrent writes to one response writer. Consume the request condition at the coordinating layer instead.
Failure boundaries
The reconciliation path reads pool state before evaluating the condition and surfaces failures instead of treating an unknown pool as empty. Cleanup errors can still follow a partial physical mutation: a failed request is not a distributed rollback guarantee. Retrying and checking actual state remain necessary after a storage failure. See multi-pool consistency.
Selection and evaluation
Evaluate once
erasureServerPools.DeleteObject holds the outer lock. Multiple pools use deleteObjectReconciled; the single-pool path reads the selected representation and invokes CheckPrecondFn before its delete-marker shortcut. Lower layers do not reinterpret the condition per copy.
Wildcards and delete markers
The DELETE helper treats * as representation existence. A delete marker does not satisfy it. A missing key or addressed version follows the corresponding not-found path; it is not manufactured into an empty ETag match.
Authorization
The handler authorizes deletion using the effective version. It does not perform the additional s3:GetObject check described in the original proposal for a specific ETag. Do not treat the proposal’s permission matrix as implemented AWS parity.
SSE-C ETags
Conditional deletion compares the established client-visible ETag projection; it does not read or decrypt the object’s payload. SSE-C read-key authentication is a separate contract from deleting an object.
Explicit versions
An explicit versionId selects that version for comparison and deletion. A matching historical ETag can therefore allow deletion of the historical version even if the current version has another ETag. This differs from the early proposal’s current-version-only rule.
Unsupported edges
An empty If-Match installs no condition. The recursive prefix-delete extension bypasses object preconditions. Batch XML ETags are not recognized by ObjectToDelete; there is no request-wide NotImplemented guard. Callers needing compare-and-delete must use the supported single-object path with a nonempty condition and verify their selected release.
Alternatives and scope
Changing a shared ETag comparator cannot fix pool selection or mutation ordering. Aggregating errors after per-pool callbacks cannot undo deletions already performed. A new transaction framework is unnecessary for consuming one callback at the existing namespace lock, but batch execution and policy enforcement require separate implementation.
Evidence and release boundary
The source evidence is the handler, pool coordinator, and batch request model. Relevant coverage includes ETag mismatch, wildcard/delete markers, explicit versions, quorum failures and multiple pools. Earlier local review or test claims for a different implementation do not establish these missing guards. A merged implementation and a released, deployed binary remain separate facts.
Follow-up work
Batch conditions
Implementing per-item ETags requires parsing them, evaluating each logical object under the correct lock, preserving quiet mode and reporting each failed condition in the per-item response. Until then, a batch ETag is ignored and must not be used as a concurrency guard.
Policy enforcement
The maintained policy package does not define s3:if-match. Supporting it requires a package change and release, correct request condition values, a Server dependency update and authorization tests. Existing If-Match execution does not by itself provide policy enforcement.
Design cost
Single-object execution reuses the established object-selection and locking boundary. Full batch conditions and policy enforcement cross additional interfaces and remain separate work. The useful invariant is narrow: a false supported single-object condition must be decided before deletion, once for the selected logical representation.
4.14 - DSN-Only Database Notifications: A Compatibility Boundary for #53
Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.
This document is the product requirements and final design record for SILO issue #53. It records the accepted compatibility boundary, implementation, and verification for PostgreSQL and MySQL bucket-notification targets.
Decision
SILO will retain PostgreSQL and MySQL notification targets, but support exactly one current configuration form for each:
- PostgreSQL requires a complete
connection_string. - MySQL requires a complete
dsn_string.
The old five-field form — host, port, username, password, and database — remains unsupported by the current KV configuration system. SILO will not re-register those keys and will not synthesize a DSN from them during legacy migration.
The legacy migration contract is deliberately narrow:
| Legacy target | Result |
|---|---|
| Disabled | Ignore it; no target is emitted. |
Enabled with a non-empty connection_string or dsn_string |
Migrate only the canonical connection-string key and the other registered target settings. |
| Enabled with only discrete connection fields | Reject migration and abort server startup before the new configuration is activated, with an actionable error that names the subsystem and target but never prints a credential. |
This is a configuration-boundary decision, not removal of the database-notification feature.
Status: implemented in f1ba68358 and included in Server 20260903.
Owner: SILO server repository.
Tracking: pgsty/silo#53.
Target: the next SILO patch release after implementation and verification.
Context
SILO inherited two generations of database-notification configuration from MinIO.
The pre-KV JSON configuration could describe a database connection either as a complete string or as five fields:
The current KV configuration exposes only the driver-native form:
This direction is not new. MinIO deprecated the five discrete fields in RELEASE.2020-04-10T03-34-42Z and instructed operators to move to connection_string or dsn_string. SILO’s current help tables, environment-variable documentation, and examples already present the complete string as the supported interface.
SILO is a new community fork with an explicit migration step. Its compatibility contract prioritizes the S3 and Admin APIs, current MINIO_* settings, on-disk data, and current KV configuration. It does not need to perpetuate every pre-2020 configuration spelling when a supported canonical form has existed for years.
The defect
Before the fix, the legacy migration helpers, SetNotifyPostgres and SetNotifyMySQL, wrote both forms into the new KV configuration. Even when the old target already had a complete connection string, the helpers also emitted all five discrete keys, usually with empty values.
The new parser rejects those keys because neither DefaultPostgresKVS nor DefaultMySQLKVS registers them. Key validation checks key presence, not whether the corresponding value is empty. Both legacy source forms therefore fail:
The failure is amplified by notification initialization. FetchEnabledTargets is fail-fast across notification subsystems: the first invalid subsystem returns an error and a nil target list. The caller logs the error and continues starting the object server, leaving healthy Webhook, Kafka, NATS, and other targets unavailable as well.
Merely returning an error from the two migration helpers does not fix that behavior. The error propagates through readConfigWithoutMigrate and initConfig, but initConfigSubsystem currently logs non-retriable configuration errors as “some features may be missing” and returns success. The server then starts without assigning globalServerConfig; notification failure is only one consequence, because region, storage class, compression, identity, and other stored settings may also be absent. The implementation must therefore carry a typed database-migration error to the startup boundary and make that error fatal. Classifying it as retriable is also wrong because the server would retry forever without any state change that could repair the configuration.
The resulting behavior is especially dangerous because object I/O still works. Operators can see a healthy S3 service while every configured event pipeline has stopped. Targets are never constructed, so delivery or later replay of events produced during the outage must not be assumed.
There is also a diagnostic-exposure issue. The unregistered password key has no sensitivity metadata and may be copied verbatim into health or diagnostic material. The registered connection_string and dsn_string keys are already treated as sensitive values.
Why the first fix was reverted
The first repair registered the five discrete keys and taught the parser to read them. That made migrated targets pass CheckValidKeys, and it appeared attractive because the target argument structures and constructors still contain code for the old fields.
It also broke the documented connection-string path.
The shared mc admin config set tokenizer discovers field boundaries by looking for registered key names. It is not fully quote-aware. Once port became a registered key, this valid input contained what looked like a second top-level field:
The tokenizer split at the port= inside the quoted value, truncated connection_string, and handed the remainder to the port parser. The command then failed with invalid port.
Under the current tokenizer, registering common words such as host, port, and password creates a direct conflict between the connection-string grammar and the top-level KV grammar. The attempted registration fix was therefore reverted. Re-registering those keys is not an acceptable solution.
Product judgment
Database notification targets are a specialized but useful capability. They provide a direct database-backed namespace view or access journal without requiring an external event bus. That remains valuable for small deployments and for users already operating PostgreSQL or MySQL.
The legacy spelling of their connection parameters has much less value. A five-field model cannot represent the useful range of driver options: TLS modes and certificates, connection timeouts, application names, Unix sockets, multi-host PostgreSQL settings, MySQL driver parameters, and future driver capabilities. Supporting both forms also creates precedence, merging, redaction, and testing questions that do not exist with one canonical value.
The complete string is the better abstraction boundary: SILO owns notification semantics, while the database driver owns connection syntax.
The product decision is therefore to keep the capability and remove the compatibility illusion. An unsupported legacy target must be rejected clearly; it must not be accepted and transformed into a configuration that later disables unrelated targets.
Goals
- Establish
connection_stringanddsn_stringas the only supported live configuration interfaces for database notifications. - Allow a legacy JSON target that already contains the canonical string to cross the migration boundary without modification to its connection semantics.
- Reject enabled discrete-only legacy targets before a partial or invalid KV configuration is activated.
- Replace the current silent runtime failure mode of #53 — healthy targets disabled while the server appears healthy — with an explicit startup-time failure that operators must resolve before the server runs.
- Ensure no migration error, log line, health report, or diagnostic bundle exposes a database password.
- Remove the ten Postgres/MySQL exceptions from the source-level unregistered-write audit.
- Make the compatibility boundary and operator remediation explicit in release and migration documentation.
Non-goals
- Supporting both DSN and discrete database fields in the current KV interface.
- Automatically synthesizing a DSN from old discrete fields.
- Rewriting the shared KV tokenizer.
- Changing
FetchEnabledTargetsfail-fast semantics in this patch. - Silently skipping an enabled database target and continuing with partial notification coverage.
- Removing PostgreSQL or MySQL notification targets.
- Deleting the legacy struct fields needed to decode and identify unsupported input. They remain on shared target argument structs that are also used by live constructors, whose discrete-field connection-string synthesis is unreachable from current KV configuration; those fields must not become supported configuration keys.
- Correcting ignored errors from the other eight legacy notification setters. Their pre-existing silent-skip behavior remains unchanged in this narrowly scoped database-migration patch and requires a separate audit and design decision.
Functional requirements
Current configuration
notify_postgresacceptsconnection_string;notify_mysqlacceptsdsn_string.- The five discrete keys remain unregistered and rejected by current configuration commands.
- Existing full strings must continue to support the database driver’s syntax, including parameters whose names contain
host,port,user,password, ordatabase. - No new public environment variables or KV keys are introduced.
- The declared legacy variables
MINIO_NOTIFY_POSTGRES_HOST/PORT/USERNAME/PASSWORD/DATABASEand their MySQL equivalents are not wired into current parsing and remain unsupported. They must not be documented as working alternatives to the complete-string variables.
Legacy migration
SetNotifyPostgresmust return without emitting a target when the legacy target is disabled.- For an enabled target,
SetNotifyPostgresmust require a non-emptyConnectionStringand write only registered Postgres keys. If both a canonical string and discrete fields are present, the canonical string wins and every discrete value is discarded. SetNotifyMySQLmust apply the equivalent rule toDSN.- Neither helper may emit
host,port,username,password, ordatabase. - A missing canonical string must return a typed or wrapped migration error identifying the subsystem and target name.
cmd/config-migrate.gomust check and propagate both helper errors. Ignoring them is forbidden.- No partially migrated configuration may be activated or persisted after either helper fails.
- Error text may name the required key and remediation, but must not include any connection-field value.
- The propagated typed migration error must abort server startup. It must not be downgraded to the non-fatal “some features may be missing” path in
initConfigSubsystem, and it must not enter the retriable-error loop. - Validation errors for a supplied canonical string follow the same startup-fatal and secrecy rules; wrapping must add target context without repeating the DSN or its components.
Original proposed error shape (illustrative, not the shipped literal):
Operator remediation
An operator encountering the error must choose an explicit remediation path. This applies both before an initial switch to SILO and when upgrading a deployment that is already running SILO: legacy migration output is not persisted, so the same old JSON source can re-enter migration on every start. A deployment that currently starts with notifications silently broken can therefore fail to start after this repair until the source configuration is corrected.
- On a compatible intermediate MinIO release, replace the old fields with
connection_stringordsn_string, verify the target, and then migrate to SILO. - Disable or remove the legacy database target, migrate the server, and recreate the target with the canonical string afterward.
- For a fresh SILO installation, create the target directly with the canonical string; no legacy migration is involved.
- For an existing SILO deployment that still reads a legacy JSON file, stop on the previous working release, back up the source configuration, then convert, disable, or remove the database target before starting the fixed release. Do not delete or rewrite unrelated configuration.
Documentation must not suggest that a discrete-only target will be converted automatically.
Availability trade-off
This decision intentionally turns one unsupported configuration from a degraded startup into a hard startup failure. The immediate availability cost is real: a server that previously served objects while all notifications were silently dead may refuse to start after the repair.
That cost is accepted because an object server that appears healthy while configured event sinks are absent creates silent, potentially unrecoverable downstream data loss. SILO is a new fork with an explicit migration boundary, and the discrete form has been deprecated since 2020. A fatal, actionable precondition is preferable to an upgrade that reports success with reduced notification coverage. The release note must make this startup behavior prominent; it must not be buried as an internal migration cleanup.
Security requirements
- The unsupported-input error must never format the legacy argument structure or its values.
- Tests must use a sentinel password and assert that it is absent from returned errors and captured logs.
- Migrated output must contain the registered sensitive connection-string key and no standalone password key.
- If a diagnostic bundle was exported from an affected deployment before this repair, operators should treat the database password as potentially disclosed and rotate it.
Alternatives considered
Register and parse the discrete fields
Benefit: preserves the old source form and uses already existing argument fields.
Rejected because: registration makes common field names visible to the shared tokenizer and corrupts quoted connection strings. It also expands the supported public configuration surface after the fields were deprecated in 2020.
Synthesize a canonical string during migration
Benefit: preserves discrete-only legacy installations.
Rejected because: it creates permanent code and test ownership for an obsolete input form, including PostgreSQL quoting, MySQL DSN formatting, socket and IPv6 behavior, defaults, and future driver drift. For a new fork with an explicit migration boundary, the benefit does not justify the continuing surface.
Skip only the unsupported target
Benefit: keeps the object server and other notification targets running.
Rejected because: silently discarding a configured event sink can cause unobservable and unrecoverable event loss. A clear migration failure is safer than an apparently successful upgrade with reduced notification coverage.
Change global notification fail-fast behavior
Benefit: limits the blast radius of future invalid targets.
Rejected for this change because: it neither repairs the database target nor closes the credential-exposure path, and it changes system-wide error semantics. It may be evaluated independently with its own operational contract.
Remove database notification targets
Benefit: removes the complete database-specific maintenance surface.
Rejected because: the targets remain useful and self-contained. The defect belongs to an obsolete configuration form, not to the notification capability itself.
Implementation scope
The server change should remain narrow:
- Update
internal/config/notify/legacy.goso the two database setters emit only canonical registered keys and reject enabled targets without a canonical string. - Update
cmd/config-migrate.goto propagate the two database-helper errors with subsystem and target context. - Define a typed database-migration error and update
cmd/server-main.gosoinitConfigSubsystemreturns it as fatal instead of logging and ignoring it. It must remain non-retriable. - Leave ignored errors from the other eight legacy notification setters unchanged in this patch; record them for a separate audit rather than expanding #53 implicitly.
- Remove all ten Postgres/MySQL entries from
knownUnregisteredWrites; the ratchet should become empty unless another independently justified legacy exception exists. - Add focused migration, startup, validation, secrecy, and coexistence tests.
- Update database-notification and migration documentation in
silo.pgsty.com.
The patch must not register the old keys, change the generic tokenizer, or refactor unrelated notification targets.
Acceptance criteria
The implementation is complete only when all of the following are demonstrated:
-
A legacy PostgreSQL target with a complete connection string migrates, passes
CheckValidKeys, and is returned byGetNotifyPostgresunchanged. -
A legacy MySQL target with a complete DSN does the equivalent.
-
Discrete-only enabled targets for both databases fail before target initialization with an actionable error containing the subsystem and target name, and server startup aborts.
-
Missing-string and malformed-string errors contain none of the sentinel host, username, password, database, or DSN values.
-
Disabled discrete legacy targets do not create configuration entries and do not block migration.
-
Migrated KVS output contains none of the ten discrete keys, including empty ones.
-
When a legacy target contains both a canonical string and conflicting discrete values, only the canonical string is migrated and no discrete sentinel appears in any output KVS value.
-
A
SetKVSregression test using the realDefaultPostgresKVSandDefaultMySQLKVSkey sets accepts a quoted connection string containingport=,host=, orpassword=. -
A configuration containing healthy Webhook, Kafka, or NATS targets cannot reach
FetchEnabledTargetswith an invalid migrated database target becausereadConfigWithoutMigratefails without yielding, persisting, or activating a partial configuration, and startup aborts on that typed error. -
initConfigSubsystemreturns the typed migration error; it neither logs-and-continues nor enters the retriable loop. -
knownUnregisteredWritesno longer contains Postgres or MySQL exceptions. -
The following verification passes:
The verbose
cmdoutput must show that tests with both prefixes actually ran; a zero-match warning is a failed acceptance check. The normal server CI suite must also pass. In the documentation checkout, runmake check.
Implementation result
Server commit f1ba68358 implements the accepted design without expanding the public configuration surface:
- the two legacy database setters emit only
connection_stringordsn_stringplus registered target settings; - disabled targets remain ignored, while enabled targets without a canonical string return a value-free
LegacyDatabaseTargetError; - only the two database migration errors are newly propagated;
- the typed error is non-retriable, escapes
initConfigSubsystem, and is classified as fatal byserverMainbeforelogger.FatalIfexits the process; - the ten Postgres/MySQL exceptions were removed from
knownUnregisteredWrites; - focused tests cover complete-string round trips, canonical precedence, discarded discrete values, secrecy, failed-migration atomicity, startup classification, and the real tokenizer key sets.
The final local Claude Code review used Claude Fable 5 at max effort and returned GO with high confidence and no blocking findings. Verification included the focused package set, race tests, go vet ./cmd, and the complete go test ./cmd -count=1 suite. The review authorized only the six-file server commit; publication remains a separate gate.
Cross-repository review found no implementation changes are required in pgsty/mc, pgsty/silo-pkg, or pgsty/silo-console: the client forwards configuration text, the package repository owns no notification schema, and Console already serializes its form into the canonical connection_string or dsn_string. The public reference and compatibility documentation is updated with this record.
Release and compatibility statement
The release note must describe this as an enforced compatibility boundary:
SILO database notification targets require
connection_stringfor PostgreSQL anddsn_stringfor MySQL. The pre-2020 discretehost/port/username/password/databaseform is not migrated. Convert or recreate such targets before switching the deployment to SILO.
Deployments already running SILO with an old-format source configuration are equally affected: after this release the server will not start until each enabled legacy database target is converted, disabled, or removed.
The issue should close only after the repair is present in a published server tag. A merged patch, a local site build, and a published release are separate completion gates.
Review record
Claude Fable 5 reviewed the first draft at xhigh effort on 2026-08-23 and returned approve with required changes. The required calibration was incorporated: startup-fatal propagation now extends through initConfigSubsystem; already-running SILO deployments are covered; the availability trade-off is explicit; canonical-string precedence, dead legacy environment variables, other ignored helper errors, and executable tests are specified.
The same model then completed a final source-backed verification pass. Final verdict: approve, with no blocking findings. It confirmed that the English and Chinese records are aligned, the requirements are implementable against the current server tree, and the acceptance criteria cover the startup, migration, parser-regression, and secrecy boundaries.
After implementation, a separate local Claude Code review using Claude Fable 5 at max effort traced the path through ExitFunc(1), inspected driver error behavior, ran the focused, race, vet, and full cmd suites, and returned GO with high confidence and no blocking findings.
4.15 - Preview Text, Never Execute It: SILO Console Text Preview PRD
Status: shipped in SILO Console 2.2.0 · Owner: pgsty/silo-console · Tracking: pgsty/silo#17 · Review: consensus of product, security, and frontend architecture reviews
SILO Console can preview images, PDFs, audio, and video, but not the small logs, text files, JSON documents, and XML documents that operators inspect every day. A correctly stored Content-Type does not help: these objects are classified as unsupported before the preview renderer is selected.
Restoring the old browser-native behavior would be easy. It would also be the wrong fix. An object in storage is controlled by the user who uploaded it. Loading that object as a same-origin HTML or XML document would turn a convenience feature into an execution boundary.
The accepted design therefore makes a stronger promise:
SILO previews eligible objects as bounded UTF-8 text. It never asks the browser to interpret their markup, MIME type, or file contents as a document.
This record fixes the product boundary, the resource limit, the security invariants, the implementation shape, and the evidence required before the feature can ship.
Decision
The first release will add a dedicated text preview type and a PreviewText component.
The contract is:
- Preserve every existing image, PDF, audio, and video classification.
- Only when the existing classifier returns
none, consider a text fallback. - Admit the four target extensions or four exact passive text MIME types.
- Fetch bytes through the ordinary authenticated download path, without
preview=true. - Enforce a hard application read limit of 1 MiB.
- Decode only strict UTF-8 and reject binary-looking content.
- Render one React text node inside a scrollable
<pre>. - Never use an iframe, HTML parser, XML parser, or HTML injection API.
- Show the complete object or no object; do not show a truncated JSON or XML document.
- Keep download available for files that are too large, invalidly encoded, or otherwise unavailable.
No new API route or S3 operation is added, and the backend inline MIME allowlist is unchanged. Delivery did change Console download responses: zero-byte Range requests return an empty 200, unsatisfiable ranges return 416 with Content-Range: bytes */N, and size is always emitted in object JSON. See Console 2.2.0.
Current behavior
The defect is present in SILO Console v2.1.1, the version currently pinned by SILO when this design was written.
The frontend preview union contains only:
Its extension table contains media formats, but not .log, .txt, .json, or .xml. Its MIME classifier likewise ignores text/plain, application/json, application/xml, and text/xml.
Runtime verification produced this split:
| Object | Frontend result | Console download response |
|---|---|---|
.log / text/plain |
none |
inline, SAMEORIGIN |
.txt / text/plain |
none |
inline, SAMEORIGIN |
.json / application/json |
“Preview unavailable” | inline, SAMEORIGIN |
.xml / application/xml |
none |
attachment, DENY |
The object-detail action also uses the wrong conjunction when deciding whether Preview should be disabled. An authorized user can click Preview for an unsupported object and receive only the unavailable message; in other combinations, the UI can offer an action before the server rejects it.
The preview component still contains a generic same-origin iframe fallback. It is unreachable under the current type union, so the current defect is not an exploitable text-preview XSS. The dead branch is nevertheless hazardous: adding text to the union and letting it fall through would reactivate precisely the document-loading behavior this design rejects.
Root cause
This is contract drift across three independently evolved layers.
Classification drift
The browser code decides eligibility from filename and object metadata, but its closed type union has no text representation. Correct metadata cannot select a renderer that does not exist.
Response-policy drift
The Console server separately decides whether a response may be inline. It still treats plain text and JSON as safe passive MIME types, while XML and HTML remain attachments. That server decision is not reflected in the frontend classifier.
Renderer drift
The old generic iframe remains after the set of reachable preview types became media-only. The code therefore suggests a capability that the type system can no longer invoke.
The repair must realign the three layers without making MIME metadata a security boundary.
Why same-origin iframe preview is rejected
X-Frame-Options: SAMEORIGIN is not a sandbox. It controls who may embed a response; it does not limit what code inside a same-origin frame can do.
If uploader-controlled HTML, XHTML, SVG, or active XML were ever served as an inline same-origin document, it could act with the Console origin. An HttpOnly cookie would prevent direct cookie reads, but it would not prevent authenticated same-origin requests. A permissive or accidentally widened MIME rule would then turn stored content into stored application code.
nosniff, Content Security Policy, and Content-Disposition remain useful defense in depth, but none replaces the core invariant:
Product contract
The feature is a read-only text viewer, not a web previewer and not an online editor.
The user should be able to:
- open a small eligible object from either the list or object-detail surface;
- read whitespace-preserving source text in the existing preview modal;
- select and copy text using browser-native behavior;
- understand whether a failure is caused by size, encoding, permission, object replacement, or network error;
- download the original bytes at any time.
The user must never be led to believe that:
- formatted JSON is the stored object;
- a partial XML document is complete;
- replacement characters are original bytes;
- an unsupported encoding has been decoded faithfully;
- an active HTML/XML document has been safely “sanitized” and executed.
Goals and non-goals
Goals
- Preview small logs, text, JSON, and XML without a local download.
- Keep object content inert regardless of extension, MIME, or payload.
- Bound retained response bytes and rendered text to 1 MiB.
- Preserve the stored text rather than silently reformatting it.
- Keep list and detail actions consistent with permissions and type eligibility.
- Support current object versions and explicitly selected historical versions.
- Preserve anonymous-access and subpath-hosting behavior.
- Ship the feature in Console first, then consume that exact Console revision in SILO.
Non-goals
- HTML or XHTML rendering.
- XML parsing, XSLT, external entities, or schema validation.
- Markdown rendering.
- JSON pretty-printing.
- YAML or CSV-specific behavior.
- Editing or saving.
- Syntax highlighting, line numbers, search, folding, ANSI rendering, or linkification.
- Head, tail, or truncated previews for large objects.
- Lossy decoding or automatic detection of GBK, UTF-16, Latin-1, or other encodings.
- A new backend text-preview endpoint.
- Changes to the existing SVG, media, PDF, download, share, or storage contracts.
An object such as notes.md may still be shown as raw text when its exact MIME type is text/plain. It does not gain Markdown semantics.
Eligibility contract
Eligibility is deliberately two-stage.
Stage 1: preserve the legacy media decision
Run the current image, PDF, audio, and video classifier unchanged. If it returns anything other than none, return that result.
This preserves historical behavior for conflicting filename and MIME combinations.
Stage 2: apply text fallback
Only after the legacy result is none:
-
Reject final extensions
.html,.htm, and.xhtml. -
Match the final filename extension case-insensitively against:
.log.txt.json.xml
-
Normalize Content-Type by removing parameters, trimming whitespace, and lowercasing it.
-
Match the normalized MIME exactly against:
text/plainapplication/jsonapplication/xmltext/xml
An allowed extension or an allowed exact MIME is sufficient. Broad matches such as text/, substring tests, and application/+json are forbidden in this release.
The resulting matrix is normative:
| Filename and MIME | Result | Reason |
|---|---|---|
report.txt + image/png |
image | Existing media decision wins. |
report.json + application/pdf |
Existing media decision wins. | |
server.LOG + application/octet-stream |
text | Allowed extension, case-insensitive. |
no extension + application/json; charset=utf-8 |
text | Exact normalized MIME. |
page.html + text/plain |
none | Explicit active-extension exclusion. |
page.txt + text/html |
text | Extension admits it; HTML source remains inert text. |
notes.md + text/plain |
text | Exact MIME admits raw text, not Markdown rendering. |
image.svg + image/svg+xml |
existing image path | No new text or iframe path. |
Filename and MIME affect product eligibility only. They never select an executable rendering mode.
Resource contract
The binary limit is:
Exactly 1 MiB is eligible. 1 MiB plus one byte is not.
Known sizes
- If the selected version has a known size greater than the limit, the initial attempt does not request its body. Explicit Retry bypasses that possibly stale listed size but retains the bounded Range and byte limit.
- A known zero size still takes the bounded request path; an empty response produces the empty-file state.
- If its known size is within the limit, begin a bounded request.
- An absent size is not the same as zero; it enters the bounded unknown-size path.
The current list-to-modal handoff must therefore preserve undefined rather than converting it to zero with a truthy fallback.
Bounded request
For a small or unknown size, request:
The extra byte is an over-limit sentinel.
The client must:
- Inspect
Content-RangeandContent-Lengthwhen present. - Read the response as a stream rather than calling
response.text()or building a complete Blob. - Retain at most the limit plus the sentinel byte.
- Cancel immediately when the sentinel byte is observed.
- Enforce the same limit when the server ignores Range and returns 200.
- Render only after end-of-stream proves that the complete object is within the limit.
An over-limit object opens an explanation state with its known size, the 1 MiB policy, and a Download action. It never shows a prefix fragment.
Request identity and cancellation
A preview request is identified by:
The request must use the existing generated API client or an equivalent base-path-safe helper so that it preserves:
- same-origin credentials;
- the current Console subpath;
version_id;- anonymous-mode
X-Anonymous: 1; - current error handling and permission boundaries.
Close, object change, version change, bucket change, and component unmount must abort the active request and clear the old content.
Abort alone is insufficient. A generation token or invalidation flag must also prevent a response that already completed reading or decoding from updating a newer preview.
An aborted request is not an error and must not produce an error toast.
Encoding and fidelity
The first release supports strict UTF-8 only:
Requirements:
- handle the UTF-8 BOM without displaying it;
- preserve Unicode text, emoji, tabs, LF, and CRLF;
- reject invalid UTF-8 rather than inserting replacement characters;
- reject decoded NUL characters as binary or unsupported content;
- do not guess another encoding;
- do not log or persist object text;
- always retain Download as the original-byte escape hatch.
The unsupported-encoding state should explain:
This object is not valid UTF-8 text or contains binary data. Download it to inspect the original bytes.
JSON and XML are displayed exactly as decoded source text. The first release must not run JSON.parse followed by JSON.stringify: that can alter unsafe integers, duplicate keys, whitespace, lexical forms, and the text users copy.
Safe renderer
The success state renders one text node:
The implementation must not use:
- iframe, object, or embed;
dangerouslySetInnerHTMLorinnerHTML;DOMParseror an XML parser;- Markdown or HTML rendering;
- an HTML data/blob URL;
- per-line or per-token spans;
- automatic links, ANSI escapes, or syntax markup.
One bounded text node keeps the DOM cost predictable and the security property inspectable.
The preformatted region uses a monospace font, preserves whitespace, defaults to no wrapping, owns both scrollbars, is keyboard focusable, and supports native selection and copy. No-wrap is intentional: it preserves aligned logs and avoids expensive layout of a single very long line.
UI states and permissions
The Preview action is enabled only when:
The object-detail conjunction bug must be fixed, and list and detail surfaces must share the same eligibility function.
An eligible over-limit object still offers Preview. The modal explains why content is not loaded; disabling the button would leave the user unable to distinguish size, permission, and type failures.
The modal distinguishes:
| State | Required behavior |
|---|---|
| Loading | Accessible busy state; no stale text. |
| Success | Scrollable raw text plus Download. |
| Empty | Explicit “File is empty” state. |
| Too large | Object size, 1 MiB limit, Download and Retry; no initial body request when size is known to exceed the limit. |
| Invalid UTF-8 / binary | Dedicated explanation and Download. |
| Forbidden | Permission-specific message; no retained text. |
| Not found / replaced | Object-change message; no retained text. |
| Network / server error | Actionable retry/download state. |
| Aborted / closed | Silent cleanup. |
HTTP error bodies must never be decoded and displayed as object content.
All new user-facing strings go through the existing translation layer and ship in English and Chinese together. The content region and controls must remain usable in light and dark themes and at narrow widths.
Functional and security requirements
Functional requirements
- FR1: Existing media and PDF classification remains unchanged.
- FR2: The text fallback follows the normative extension/MIME matrix.
- FR3: Eligible complete objects up to 1 MiB render as strict UTF-8 source.
- FR4: Over-limit objects render no partial content.
- FR5: Empty objects have a distinct successful empty state.
- FR6: Current and selected historical versions use the same version for metadata, size, and body.
- FR7: Anonymous access and subpath hosting retain their current request behavior.
- FR8: List and detail actions apply the same type and permission decision.
- FR9: Download, share, media, PDF, and storage behavior do not change.
Security requirements
- SR1: Object bytes can reach the DOM only through text content.
- SR2: Text Preview contains no document renderer or parser.
- SR3: At most 1 MiB plus one sentinel byte is retained.
- SR4: Closing or changing identity invalidates every previous response.
- SR5: Invalid UTF-8 and NUL content are not shown as faithful text.
- SR6: Errors, Redux, local storage, logs, and telemetry never retain preview text.
- SR7: Server authorization remains authoritative for direct requests.
- SR8: No CSP or backend inline MIME relaxation is introduced.
Implementation scope
Expected Console changes:
- Refactor preview classification so the current media decision is preserved and text is an explicit fallback.
- Add
textto the preview type union. - Add a dedicated
PreviewTextcomponent with streaming bounds, strict decode, request cancellation, and explicit states. - Route text objects explicitly to that component.
- Remove the unreachable generic iframe fallback.
- Fix the object-detail Preview disable expression and share eligibility logic with the list surface.
- Preserve unknown size instead of coercing it to zero.
- Add English and Chinese strings.
- Add classification, component, resource, security, permission, version, and browser tests.
Expected unchanged areas:
- Console and S3 API paths;
- the backend
safeMimeTypeslist; - Content Security Policy;
- object storage and metadata formats;
- image, PDF, audio, video, download, and share handlers;
- external frontend dependencies.
If a future product requires tailing, server-side transcoding, organization-wide policy, or reliable behavior through proxies that ignore Range, a dedicated server endpoint may be designed separately.
Rejected alternatives
Keep text preview disabled
Benefit: no new code or browser memory use.
Rejected because: logs and configuration objects are a routine object-storage workflow, and download-only inspection is an avoidable Console regression.
Reuse the same-origin iframe
Benefit: minimal code and browser-native presentation.
Rejected because: it turns uploader-controlled content and mutable MIME metadata into a same-origin document boundary. It also leaves resource use unbounded.
Add a backend preview API now
Benefit: central server-side limits and normalized text responses.
Rejected for the first release because: the user already has object-read permission, and the existing download endpoint provides versioning, authorization, and Range. A new API would duplicate contracts without establishing a new data-access boundary.
Show the first 1 MiB of a large object
Benefit: better large-log convenience.
Rejected because: partial JSON/XML is structurally misleading, UTF-8 boundaries need additional handling, and a single “preview” action would no longer mean complete content.
Decode invalid UTF-8 with replacement characters
Benefit: some damaged or legacy logs remain partially readable.
Rejected because: copied text would no longer faithfully represent the stored object. Lossy viewing and other encodings require a separate, explicit product mode.
Auto-format JSON
Benefit: more readable indentation.
Rejected because: parse/stringify can alter numbers, duplicate keys, lexical representation, and copied content. A future opt-in formatted view may sit beside, never replace, the raw default.
Add Monaco or another code editor
Benefit: line numbers, search, highlighting, and folding.
Rejected because: bundle, worker, CSP, and maintenance costs exceed the needs of a bounded read-only preview. A native <pre> is smaller and easier to audit.
Acceptance and test plan
Classification matrix
Automated tests must lock every normative matrix row, extension case handling, MIME parameter stripping, explicit HTML/XHTML denial, and unchanged media conflicts.
Resource tests
Cover:
- 0 bytes;
- 1 byte;
- exactly 1,048,576 bytes;
- 1,048,577 bytes;
- known over-limit size with zero body requests;
- unknown size;
- 206 with a revealing
Content-Range; - server ignores Range and returns 200;
- missing or false
Content-Length; - close and identity changes during streaming.
No case may retain or render more than the complete allowed object.
Encoding and fidelity tests
Cover UTF-8 Chinese, emoji, tabs, LF, CRLF, BOM, invalid byte sequences, NUL bytes, JSON unsafe integers, duplicate keys, original whitespace, XML declarations, DOCTYPE, CDATA, and stylesheet processing instructions.
The raw success view must preserve decoded text. Invalid and binary cases must show their dedicated state.
Security tests
Payloads containing <script>, event attributes, iframe tags, SVG handlers, XML stylesheets, external entities, and suspicious URLs must:
- appear literally in
<pre>.textContent; - create no corresponding DOM elements;
- execute no script or dialog;
- cause no object-content-originated request;
- encounter no iframe, object, embed, HTML parser, or XML parser in Text Preview.
Permission and race tests
Verify:
- no
GetObjectmeans no usable action and no retained body; - historical versions require their corresponding permission;
- metadata and body use the same version ID;
- a late old response cannot replace a new object’s preview;
- 401, 403, 404, 416, and 5xx bodies never become preview content;
- anonymous access and Console subpaths do not regress.
Browser regression
Use a real SILO/Console test instance to inspect both English and Chinese routes, light and dark themes, and narrow and desktop widths. Media, PDF, download, share, and version workflows require smoke coverage alongside the new text states.
Delivery and completion gates
The change belongs to pgsty/silo-console, even though the user report is tracked in the SILO server repository.
Delivery is staged:
- Merge the focused Console source and test change.
- Pass TypeScript checking, production build, automated matrices, and real-browser security regression.
- Update Console release notes and regenerate the actual embedded web assets.
- Publish a Console version; a minor release is appropriate for the new visible capability.
- Update SILO’s
github.com/minio/console => github.com/pgsty/silo-consolereplacement to the exact new pseudo-version. - Build a SILO candidate from that exact dependency and repeat integration checks.
- Publish the SILO binary and image, naming the first version that contains the feature.
These are separate states:
| Gate | Meaning |
|---|---|
| Console PR merged | Implementation exists in source. |
| Console assets/tag published | Console is independently consumable. |
| SILO dependency updated | SILO main has integrated the change. |
| SILO release published | Users can obtain the feature. |
Issue #17 should not be described as fixed for users merely because a local preview or Console source PR exists.
Trade-off summary
The accepted design favors:
- explicit scope over a generic browser viewer;
- complete small files over partial large files;
- source fidelity over automatic formatting;
- strict UTF-8 over silent lossy decoding;
- one inert text node over a full editor;
- the existing download API over a new backend contract;
- a verifiable security invariant over convenient same-origin rendering.
The cost is real: large logs and legacy encodings still require download, and the first release has no search, line numbers, wrapping toggle, or highlighting. Those omissions are deliberate. They make the feature small enough to audit and strong enough to trust.
Review record
The design was independently reviewed from three perspectives:
- product scope, delivery, and acceptance;
- security and frontend architecture;
- compatibility and current-source verification.
The reviewers initially differed on MIME-only eligibility and lossy UTF-8 fallback. After cross-review they reached a single contract:
- existing media classification wins;
- text fallback accepts the four target extensions or four exact normalized MIME types;
- HTML/XHTML extensions are explicitly excluded;
- strict UTF-8 and NUL rejection are required;
- lossy viewing is deferred to a separate proposal.
No unresolved design question remains. Implementation may proceed against this record.
Current Retry boundary: the too-large state permits reprobe of potentially stale listed size, never an unbounded download. The Range, response-header checks and at-most-1 MiB + 1-byte read limit still apply. Empty files are verified through the response path; unknown size must not be treated as zero.
4.16 - Go 1.27 TLS Defaults and OIDC Discovery Failure Modes
After SILO’s toolchain moved to Go 1.27, the Server TLS repair
48e184652
(“fix(tls): honor Go key exchange defaults across transports”) removed its
explicit curve overrides. This page records the TLS changes and the diagnostic method for OIDC
discovery failures that came out of issue #154,
and the operational facts an administrator needs when identity goes missing
at startup.
Release boundary, 2026-09-16: Server 20260903 already uses Go 1.27.1, but does not contain
48e184652. That later TLS repair is on main; upgrading the compiler and adopting this repair are separate changes.
Evidence class, stated up front. Every mechanism below is verified by synthetic experiments: ClientHello captures, fresh-process CA probes, and fixture reproductions. The #154 customer’s discovery URL and ingress configuration were never obtained, so no root cause is claimed for that deployment — two locally verified mechanisms could each produce the reported symptom, and either the ingress rejecting the new handshake, or a proxy rejecting the changed User-Agent, remains plausible. #154 stays open for an affected-environment retest.
What Go 1.27 changed
- Explicit curve preferences now override the ML-KEM compat switches.
GODEBUG=tlsmlkem=0removes all ML-KEM hybrids from the default set;tlssecpmlkem=0removes only the P-256/P-384 hybrids introduced in Go 1.26 and retains X25519MLKEM768. An application that configuresCurvePreferencesexplicitly keeps ML-KEM in whatever list it names — a deliberate Go 1.27 change. SILO had eight TLS configuration points setting an explicit list including X25519MLKEM768; the fix removes all eight assignments and retires the helper, so these Server configuration points follow Go’s defaults and the compat switches work again. The stack review found pkg, mcli, and Console clients already used defaults; Console’s HTTPS listener retains its separate P-256-only policy. - ClientHello now offers ML-DSA signature algorithms (identifiers
0x0904–0x0906). ML-DSA is a signature scheme and distinct from ML-KEM: disabling hybrid key exchange does not disable the ML-DSA offer, and an ingress that rejects ML-DSA is not fixed by any ML-KEM switch. - The ClientHello grew. Measured on the same source and dependencies:
Go 1.26.5 default 1497 bytes; Go 1.27.1 default 1509 bytes; the old
explicit list under
tlsmlkem=0produced a 275-byte hello with no ML-KEM, while Go 1.27.1 with an explicit list still produced 1509 bytes containing ML-KEM. Rebuilding with a different compiler alone changed the handshake. - macOS root-CA behavior flips with the module’s Go directive. A fresh
process honoring
SSL_CERT_FILE/SSL_CERT_DIRinstead of the Keychain is governed by thex509sslcertoverrideplatformGODEBUG default, which follows the main module’sgodirective:go 1.26modules ignore those variables on macOS (platform store wins),go 1.27modules honor them — and the consuming application’s directive wins even when a library module is older. Operators on macOS should know that setting either variable replaces Keychain trust wholesale with the file/directory given; a stale or incomplete path then breaks chains the Keychain would have accepted, and unsetting restores the Keychain. - Not everything changed. TLS versions, cipher suites, certificate and hostname verification, proxy handling, and HTTP/2 selection are unaffected. The standard-library drain cap (256 KiB / 50 ms) and other audited Go 1.27 changes showed no SILO dependency. Go 1.27 binaries require macOS 13 or newer. Downgrading is not a supported path: the module graph requires Go ≥ 1.27.1 across Server, Console, and mc.
Why the OIDC-only patch was withdrawn
The investigation first produced a minimal candidate: clear
CurvePreferences on the OIDC discovery transport only. It was deliberately
not shipped. The same transport serves identity plugins, notification and
lambda reachability checks, audit webhooks, and S3 cloud-backend tiers —
fixing two OIDC call sites would have left every other consumer on the
defective explicit list. The merged repair removes the explicit curves at all
eight Server configuration points so the compat switches apply there, keeps
certificate verification strict, and adds no protocol downgrade or automatic
fallback. The archived one-transport patch must not be reapplied on top of
the merged fix.
Diagnosing a discovery failure by phase
The startup chain is: server start → identity system init → fetch
.well-known/openid-configuration (discovery) → fetch the jwks_uri keys →
IAM store ready → Console initializes. Console’s own OIDC configuration
dialog validates through the same server-side transport. A failure anywhere
in the chain leaves IAM offline; a successful discovery does not clear the
JWKS fetch, and a 503 on JWKS blocks IAM just as hard.
Discriminate by where the connection dies:
- Reset right after the TLS ClientHello (
tls_startthen reset): suspect the ingress’s ClientHello handling — proxy CONNECT rules, TLS terminators, or anything keyed on hello size or contents. This is where the Go 1.27 changes land. - Reset after TLS completes (
wrote_requestthen reset): the TLS layer is fine; look at HTTP-layer policy — WAF rules, User-Agent allowlists (the server’s UA changed fromMinIOtoSilowith the rebrand), routing. Replacing certificates or key exchange here has no targeted effect. - x509 errors: compare the chain actually received, the SNI, and the trust store the process resolves (see the macOS section above).
- Always test from the same network position as the failing process — a fresh container does not inherit the failing container’s network namespace, and same-IP/same-proxy controls come first.
The health endpoint that tells the truth
/minio/health/live and /minio/health/ready both stay 200 while IAM is
offline — readiness as deployed does not cover the identity system. The
endpoint that reports it is /minio/health/cluster, which checks identity
initialization and returns 503 with the X-Minio-Server-Status: iam-offline
marker. Monitoring that should catch a broken IdP integration should probe
the cluster health, plus one authenticated operation.
Recovery is automatic: identity initialization retries at randomized 0–3 s intervals, and a recovered IdP brings IAM back without a restart (observed sub-second to ~1.4 s locally). Retrying cannot fix a persistent incompatibility — a hello the ingress rejects stays rejected.
Transport facts worth knowing
The discovery/JWKS client builds its own transport: HTTP/2 disabled (no ALPN,
HTTP/1.1), proxies taken only from HTTPS_PROXY/NO_PROXY (uppercase
preferred; ALL_PROXY unused), DNS refresh defaulting to 30 s in Kubernetes/Docker and 10 min otherwise (overridable by the DNS cache TTL setting), dialing that walks
addresses in order without shuffling, and timeouts of 5 s per TCP dial,
10 s for the TLS handshake, and 1 min to response headers. There is no
total timeout on the discovery or JWKS fetch itself — a slow IdP can hold
startup indefinitely; tightening that is a known, separately-sized follow-up.
Attribution
This record distills the issue #154 investigation and the September Go 1.27 stack review; the reproduction artifacts and the full evidence chain are retained outside the documentation tree. The supported statement is: the merged fix restores Go key-exchange defaults at the eight affected Server configuration points, verified with synthetic negative controls — it does not claim to have diagnosed any specific hidden deployment, and #154 remains open pending a retest in the affected environment.
Go behavior is grounded in the official 1.27 release notes and the actual toolchain. ClientHello byte counts above describe the recorded fixtures, not a fixed size for every connection.
4.17 - No I/O Before Auth, No Privilege From Headers
Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.
This record describes the CORS hot-path and replication-request trust repair merged into SILO as PR #101 (938603458 through 04b097fd9).
Status on 2026-09-03: PR #101 merged into
mainon 2026-09-01 with four follow-up commits: per-entry Snowball trust isolation (ff44527a3), request defaults preserved across Snowball workers (ab3ae99ca), and replication validity probes that verify the replication permissions (c9ad74673) under the rule prefix (5db7be4ee). Implementation, focused and race tests, the complete server package suite, object-lock tests, vet, build, two rounds of Fable 5 design review, repeated Opus 5 adversarial acceptance, and a real local TLS two-site replication run are complete. The pre-release cleanup kept the resident-only lookup with its fail-closed startup and load-failure states, dropped only the internal-namespace special case, and made the header-stripped request clone share the original request trailer so streaming-checksum uploads keep working for untrusted requests. Tag, package, image, deployment, and production verification remain separate gates.
Scope: HTTP request interpretation before and inside the S3 handlers. No S3 wire field, object format, bucket metadata format, replication protocol, encryption format, or client command changes.
Security properties: pre-authentication CORS processing performs no object-layer I/O; a header never grants replication semantics by itself; SSE-C ciphertext paths and replica-only metadata require both authentication and the corresponding replication permission.
Too Long; Didn’t Read (TL;DR)
Two bugs looked unrelated:
- an
Originheader made the outermost CORS middleware treat the first URL segment as a bucket and synchronously load its metadata before authentication; X-Minio-Source-Replication-Requestmade downstream code believe a request was internal replication merely because the header existed.
They shared the same design failure: untrusted request shape was allowed to acquire expensive or privileged internal meaning before an authorization boundary.
The repair establishes two invariants:
For CORS, the outer middleware now reads only metadata already resident in memory. For replication, handlers authenticate the original signed request first, authorize the appropriate replication action, and then attach a private trust decision to the request context. Untrusted internal headers are stripped only after signature verification. The context decision—not header removal—is the authority used by option builders, encryption paths, object lock, event generation, and metadata persistence.
Failure A: pre-authentication CORS amplification
corsHandler wraps the complete server router. Any request carrying Origin reaches it before S3 authentication, request validity checks, and the normal API limiter.
The per-bucket CORS implementation originally called the normal bucket metadata getter:
When .metadata.bin did not exist, the loader intentionally searched legacy configuration files. With none found, it returned a valid empty metadata record rather than NoSuchBucket. The generic getter then inserted that record into metadataMap.
An unauthenticated client could therefore vary otherwise plausible names and obtain two effects per distinct value:
- repeated erasure/object metadata reads before the normal request limiter;
- growth of the in-memory bucket metadata map.
Name validation alone cannot repair this. An attacker can generate an effectively unbounded sequence of syntactically valid, nonexistent bucket names. Distributed deployments eventually prune stale map entries during the 15-minute metadata refresh; single-node deployments do not start that refresh loop, so their synthetic entries persist until restart.
Failure B: a marker header became authority
SILO and its MinIO-compatible clients use internal headers to preserve source state during replication. The most important marker is:
Before this repair, several paths treated header presence—or its raw string value—as proof that the request was a replication request. That affected more than metadata extraction:
GETof an SSE-C object could setNoDecryptionand return ciphertext without the customer key to a caller holding only ordinary read permission;- source ETag and modification time could replace server-generated values;
- source tagging, retention, and legal-hold timestamps could enter last-writer-wins comparisons;
- a past object-lock retention date could be accepted through a raw marker check;
- delete-marker identity and modification time could be supplied by the caller;
- successful object events could be suppressed;
- multipart actual size and encrypted checksum metadata could be injected at completion;
X-Amz-Replication-Statuscould be persisted from ordinary PUT, COPY, or POST-policy metadata extraction.
The earlier CVE-2026-34204 repair correctly stopped ordinary PUT and COPY from importing the replication SSE metadata that could make objects unreadable. It did not yet provide one authority shared by every reader of the marker, source fields, event state, object-lock exceptions, or multipart completion metadata.
Selected design
One exact marker, two trust levels
The marker is accepted only when it appears exactly once and its value is exactly lowercase true. Duplicate values, mixed case, and any other value are untrusted.
The handler then derives two related decisions:
| Decision | Requirements | Semantics it may enable |
|---|---|---|
trusted |
original request authenticated; non-anonymous principal; exact marker; s3:ReplicateObject or s3:ReplicateDelete on the addressed resource |
source ETag/MTime and source timestamps; actual size and encrypted checksum transfer; event and re-replication suppression; replication delete pool/version pinning |
replicaTrusted |
trusted, plus raw request status REPLICA or a multipart upload whose stored status is REPLICA |
replica status persistence; replication SSE sealed-key import; SSE-C ciphertext/no-decryption path; replica-only object-lock behavior |
The split is required by the real wire protocol. Not every legitimate replication request repeats X-Amz-Replication-Status: REPLICA.
The receiver follows this matrix:
| Incoming shape | Result |
|---|---|
| no marker | ordinary S3 operation |
| marker without replication permission | internal fields ignored; operation continues with ordinary semantics |
REPLICA without replication permission |
403 AccessDenied |
exact marker + replication permission, no REPLICA |
trusted only |
exact marker + replication permission + REPLICA |
trusted and replicaTrusted |
The explicit 403 for an unauthorized REPLICA request prevents a claimed replica write from being silently downgraded into a new ordinary object that may be replicated again.
Authenticate the original, then sanitize
SigV4 signs request headers. Removing an internal header before authentication would change the canonical request and turn a valid signature into SignatureDoesNotMatch.
The ordering is therefore mandatory:
The audit logger retains the original request. The effective request clone retains public S3, SSE, checksum, object-lock, copy-source, proxy, and replication-validity headers. It strips only internal source/replication controls, including source ETag/MTime/delete-marker/timestamps, replication SSE state, actual object size, encrypted checksum transfer, and the request use of X-Amz-Replication-Status.
Header stripping is defense in depth. All privileged consumers use the private context decision or an explicit Boolean; they do not infer trust by looking at the clone.
Replica status is not generic user metadata
X-Amz-Replication-Status is an S3 response header that MinIO-compatible servers also use as an internal request control. It no longer belongs to the generic supported-request-metadata list.
Ordinary PUT, COPY, multipart initiation, Snowball/PAX extraction, and POST policy cannot persist it merely by submitting the field. The receiver sets REPLICA explicitly only in a replicaTrusted branch.
This closes a subtle POST-policy path: a form field could previously store REPLICA, causing the resulting object to evade normal replication scheduling even though the POST principal never held replication permission.
Object lock receives an explicit decision
The object-lock parser used to accept past retention dates when the raw marker header was present. That package now receives allowPastRetainDate explicitly from replicaTrusted state.
The surrounding handler also uses the same decision when deciding whether an existing compliance/legal-hold version may be overwritten by a replica. This removes an internal-header dependency from the reusable object-lock package.
Actual replication wire matrix
The design was checked against the silo-go v7.3.1 emitter selected by the server’s go.mod, not inferred from comments or upstream documentation.
| Operation | Marker | REPLICA on this request |
Receiver decision |
|---|---|---|---|
regular replicated PutObject |
yes | yes | replicaTrusted |
replicated NewMultipartUpload |
yes | yes | persist trusted multipart replica provenance |
replicated PutObjectPart |
yes | no | trusted; replicaTrusted only when stored MPU status is REPLICA |
replicated CompleteMultipartUpload |
yes | no | trusted; preserve source ETag/MTime, actual size, and encrypted checksum |
| CopyObject metadata replication | yes | yes | replicaTrusted |
replicated RemoveObject |
yes | yes | replicaTrusted with s3:ReplicateDelete |
| batch replication PUT/Complete | yes | no | trusted; target credentials must hold s3:ReplicateObject |
| proxy/readiness/validity probes | separate probe headers | no marker authority | probe behavior retained; those headers are never stripped by this repair |
s3:ReplicateDelete is the trust gate, not the receiver’s only permission.
For compatibility with deployed target policies, a trusted replication delete
also requires s3:DeleteObject; an explicit deny on
s3:DeleteObjectVersion still blocks a named-version purge. Ordinary clients
do not use this compatibility path: an explicit UUID or versionId=null
requires an allow for s3:DeleteObjectVersion.
Requiring REPLICA for every trusted operation would break PutPart, multipart completion, and batch replication. Trusting every marker would recreate the vulnerability. Stored multipart provenance bridges the two requirements for encrypted raw parts.
CORS resident-only state machine
The outer CORS middleware must remain cheaper than the request it is about to route. It now calls a dedicated resident-only getter that takes one read lock and examines only in-memory state.
| Bucket metadata state | CORS result | Object-layer work |
|---|---|---|
| resident, valid per-bucket CORS | apply per-bucket rule; a failed refresh keeps the last loaded document, as for every other bucket configuration | none |
| resident, no CORS document | use global CORS fallback | none |
| resident, invalid stored CORS | fail closed; continue without CORS headers and log once | none |
| not resident while startup loading is still running | fail closed | none |
| not resident after startup: real bucket whose metadata failed to load | fail closed | none |
| not resident after startup: reserved, invalid, internal, or unknown name | global fallback | none |
The lookup consults the resident map and a bounded set of real buckets whose metadata failed to load at startup or during a refresh. That set is filled only from disk-derived bucket lists, never from a client path, never records a bucket that is already resident, and a successful load, Set, bucket removal, stale-bucket reconciliation, and subsystem reset clear it. Both non-resident states fail closed: a presigned URL is authenticated by its own signature, so the bucket’s CORS document is the only origin boundary a browser enforces for it, and answering with the global policy would let a leaked URL be used from any origin. The internal .minio.sys namespace no longer has a special case; like any reserved or invalid name it is not a bucket, gets the global fallback, and is rejected downstream.
Alternatives rejected
| Alternative | Why it was rejected |
|---|---|
| Validate bucket names before the old CORS getter | valid nonexistent names still provide an unbounded attacker-controlled key space and still trigger pre-auth I/O |
Call GetBucketInfo before loading CORS |
replaces eleven metadata reads with at least one unthrottled backend operation per attacker name |
| Cache every negative result with a TTL | bounds duration, not attacker cardinality or the initial I/O amplification |
| Strip replication headers before authentication | breaks SigV4 canonical-request verification |
| Reject every request carrying an internal marker | turns formerly ignored extra headers into broad client failures and breaks legitimate marker-only replication calls |
Require REPLICA on every trusted call |
breaks replicated PutPart, CompleteMultipartUpload, and batch replication wire behavior |
| Let every handler re-check raw headers independently | recreates inconsistent trust rules and leaves future consumers easy to miss |
Store a Boolean in ObjectOptions but leave events/object lock on headers |
produces two authorities that can disagree; the original bug class remains |
Implementation boundary
The selected change is intentionally layered:
- a small request-trust module defines exact marker parsing, replication authorization, private context state, and the post-authentication effective request;
- object option builders parse source fields only when their caller provides trusted state;
DecryptObjectInfo, event request parameters, multipart completion, delete options, and object lock consume the same decision;- handlers calculate trust immediately after their existing authentication path;
- multipart part handling combines current-request trust with stored MPU replica provenance;
- generic metadata extraction does not accept replica status;
- CORS middleware uses a separate resident-only metadata accessor and never calls the load-on-miss getter.
No object-layer API needs to infer HTTP trust. Programmatic internal callers that construct ObjectOptions{ReplicationRequest: true} remain unchanged.
Verification and adversarial review
Regression coverage includes:
- hundreds of distinct valid missing bucket names, both actual and preflight CORS requests, with zero metadata reads and no map growth;
- Console, reserved, invalid, startup, internal namespace, and invalid stored CORS paths;
- least-privilege SSE-C GET, HEAD, and GetObjectAttributes callers with correct, missing, wrong-case, and unauthorized markers;
- marker-only batch-style PUT preserving source ETag/MTime only with
s3:ReplicateObject; - unauthorized
REPLICAPUT and DELETE returning403; - POST policy unable to forge replica status;
- object-lock past-date parsing with and without replica trust;
- marker-only CopyObject with SSE-C source headers copying plaintext rather than ciphertext;
- fake marker on an ordinary SSE-C MPU failing instead of storing raw bytes;
- a real in-process SSE-C multipart replication chain: encrypted source, raw ciphertext part, trusted replica initiation, marker-only PutPart and Complete, and exact plaintext recovery with the original key.
The final local tree passed focused and race tests, the complete cmd suite, object-lock tests, vet, build, and diff checks.
A separate black-box run started two TLS-enabled SILO instances built from the candidate and enabled real site replication. It verified:
- an SSE-C 4 KiB object;
- an SSE-C 12 MiB, three-part multipart object;
- an SSE-C CopyObject result;
- a replicated delete marker.
Source and target ETag, size, version ID, SSE-C key MD5, decrypted SHA-256, and delete-marker version ID matched; targets reported REPLICA.
Two Fable 5 review rounds first corrected the trust model for marker-only batch and multipart calls, then audited the implementation. A final independent Claude Code Opus 5 review reported GO, with no P0/P1 findings, and independently reran build, vet, race, object-lock, and full cmd tests.
Compatibility and operations
- Ordinary clients: no request change. Untrusted internal headers are ignored instead of acquiring internal semantics.
- Unauthorized claimed replica writes: requests carrying
X-Amz-Replication-Status: REPLICAnow return403where some multipart subpaths previously lacked a uniform check. - Batch replication: destination credentials must include
s3:ReplicateObject, as documented in the batch replication requirements. Without it, the receiver processes marker-only writes as ordinary writes and does not preserve source ETag/MTime. - SSE-C: ordinary reads still require the customer key. Authorized replica reads may use the raw ciphertext path needed to preserve encrypted bytes.
- Events: only trusted replication suppresses replica creation/access events; a forged marker no longer silences them.
- Object lock: replica exceptions are permission-derived rather than header-derived.
- Performance: CORS removes pre-authentication backend work. Trusted writes add policy checks already required by the replication contract; no additional object pass is introduced.
- Rolling upgrade: wire and storage formats are unchanged. New receivers enforce the trust boundary; old receivers remain vulnerable to the old header semantics until upgraded. Per-bucket CORS behavior can therefore differ by node during the rolling window.
- Rollback: data written by the repaired version remains readable by the previous version, but rollback reopens both trust defects and restores pre-authentication metadata loads.
Residual risks and follow-ups
-
2026-09-09 replication reliability follow-up: Delete completion, MRF visibility, and resync cancellation records the reproductions, minimal fixes, Fable review, and PR #162 validation for #153, #152, and #137. It addresses reliability after trusted requests enter the replication pipeline, preserving this page’s authorization boundary.
-
Emit a rate-limited diagnostic when a marker-bearing request lacks replication permission; the safe ordinary fallback is otherwise easy to misdiagnose as an ETag/MTime mismatch.
-
Replication validity probes now verify the replication permissions the target credentials need and place the synthetic validation key under the rule prefix (
c9ad74673,5db7be4ee). -
This review covers the named source/replication headers. Other future internal controls must still answer the same question: which authenticated decision allowed this client value to acquire internal meaning?
Conclusion
An internal-looking header is still client input. A bucket-shaped URL segment is still attacker input. The durable repair is to stop either one from becoming authority by accident:
Before authentication, do no backend work. After authentication, derive trust once and pass the decision—not the claim—downstream.
That rule is broader than CORS or replication. It is the boundary future SILO handlers should preserve whenever inexpensive public request syntax meets expensive or privileged internal state.
This repair is SN-2026-008 in the ledger, with delivery in the 20260903 chronicle. The silo-go reference above describes the original temporary dependency. After 0079723d3, Server returned to verified upstream minio-go; the component matrix records selected versions.
4.18 - One Endpoint, Two Privileges: Separating User and Group Status
Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.
This document records the discussion, repair, and final authorization design for upstream issue minio/minio#21478 and SILO PR #73.
Status on 2026-08-26: SILO PR #73 was merged as
2e2377d1c, preserving the signed-off repair commit58735ee38. All eight reported checks passed. Upstream issue #21478 and PR #21482 remain open, butminio/miniois archived and read-only, so no further issue comment or merge can be made there.
Group follow-up on 2026-08-28: final release review found the same fixed-action defect inset-group-status. Signed-off server commit229fe2b3cnow selectsadmin:EnableGrouporadmin:DisableGroupfrom the requested target state and adds a real four-way IAM authorization test. Local verification and independent review are complete; it was merged intomainon 2026-08-29, and tag and delivery remain pending.
Scope: authorize enabling and disabling a user with their respective existing Admin Actions. Do not change the route, status values, account storage, replication record, or client API.
Security property: possessingadmin:DisableUsermust not grant the ability to enable an account, and possessingadmin:EnableUsermust not grant the ability to disable one.
Release boundary: merge, tag, release package, container image, deployment, and production verification remain separate gates.
Too Long; Didn’t Read (TL;DR)
SILO exposes both admin:EnableUser and admin:DisableUser, but the shared set-user-status handler historically authorized every request with admin:EnableUser. A policy that granted only admin:DisableUser therefore could not disable an account. The workaround was to grant admin:EnableUser as well, which destroyed the least-privilege boundary that the two action names promised.
The selected repair derives exactly one required action from the requested target state before authorization:
| Requested status | Required action |
|---|---|
enabled |
admin:EnableUser |
disabled |
admin:DisableUser |
| invalid or unknown | admin:EnableUser, preserving the previous authorization-before-validation default |
The handler then calls validateAdminReq once. A four-way IAM test proves both positive operations and both denied cross-action operations. This is intentionally stricter than preserving the accidental historical behavior in which an Enable-only policy could also disable users.
The same rule now applies to group status:
| Requested group status | Required action |
|---|---|
enabled |
admin:EnableGroup |
disabled |
admin:DisableGroup |
| invalid or unknown | admin:EnableGroup, preserving the previous authorization-before-validation default |
Before the follow-up, an EnableGroup-only principal could disable a group, while a DisableGroup-only principal received AccessDenied for that exact operation. The group repair uses the same one-selector, one-authorization design rather than treating the two actions as aliases.
The reported defect
The Admin API uses one route for both state transitions:
Before the repair, the handler checked one fixed action before reading the requested status:
The later call to SetUserStatus correctly received either enabled or disabled, but authorization had already treated both as Enable operations. admin:DisableUser existed in the policy vocabulary and documentation while being ineffective for this endpoint on its own.
Issue #21478 supplied the practical counterexample: an operator wanted a policy that could disable accounts during an incident without being able to restore them. A policy containing admin:DisableUser received AccessDenied; adding admin:EnableUser made the request work, but also gave the operator the more powerful recovery transition that the policy intentionally withheld.
This is not a missing convenience permission. It is a mismatch between the policy model and the enforcement point:
Why two actions must mean two capabilities
An account state transition has direction. Disabling is commonly delegated to incident responders, fraud controls, compliance automation, or a break-glass process. Enabling restores access and may require a separate approver.
If either action authorizes both transitions, a policy author cannot express that separation. The server would publish two names while enforcing one combined capability. The design contract is therefore strict:
| Principal policy | Disable target | Enable target |
|---|---|---|
admin:DisableUser only |
allow | deny |
admin:EnableUser only |
deny | allow |
| both actions | allow | allow |
| neither action | deny | deny |
The built-in consoleAdmin policy grants admin:*, so full administrators retain both operations. The compatibility impact is limited to custom restricted policies that relied on the old accidental behavior.
The public PBAC reference now states the same contract for admin:EnableUser and admin:DisableUser.
Design goals and non-goals
Goals
- Make both existing Admin Actions enforceable according to their names.
- Preserve least privilege in both directions.
- Perform one authorization decision and write at most one authorization error.
- Preserve the route, request values, response format, self-mutation guard, IAM storage call, and site-replication hook.
- Encode the contract in tests that fail if the two permissions are broadened or swapped again.
Non-goals
- split the endpoint into separate enable and disable routes;
- add a new combined action or change policy syntax;
- change user status persistence or replication;
- redesign Console permissions;
- infer release, image, deployment, or production delivery from a source merge.
Alternatives considered
Keep checking admin:EnableUser for both states
This preserves behavior but leaves admin:DisableUser unusable and forces over-privileged policies. It is the defect, not a compatibility contract worth retaining.
Require both actions for either transition
This makes the two labels decorative and prevents delegated disable-only operation. It is stricter in quantity but weaker in expressiveness and least privilege.
Try Enable authorization, then retry Disable authorization
Upstream PR #21482 attempted this shape for a disabled request. It first called validateAdminReq with EnableUser, then called it again with DisableUser if the first result was nil.
That helper has an important contract: when it returns a nil object layer, it has already written an error response. A Disable-only request can therefore commit a 403 response before the second authorization succeeds and the handler proceeds to mutate account state. Authorization fallback must never continue after an error response has been committed.
Accept either Enable or Disable for a disabled request
validateAdminReq already accepts multiple actions and succeeds if any one is allowed, so compatibility behavior could be implemented safely with one variadic call. That would let Disable-only policies work while preserving the historical ability of Enable-only policies to disable.
SILO rejected this option because the historical ability was the enforcement bug. It would solve the reporter’s positive case but retain a cross-action privilege that contradicts the two-action model. Operators who want both transitions can grant both actions explicitly.
Validate the status before authenticating
Rejecting unknown status values first would change error precedence: a caller that previously had to pass the Enable authorization gate could now receive a validation result before authorization. The repair does not need that broader behavioral change.
Unknown values therefore retain admin:EnableUser as the authorization default. Valid disabled is the only value that selects admin:DisableUser; the existing IAM layer remains responsible for rejecting invalid status values after authorization.
The selected implementation
The repair adds a pure selector:
The handler reads the route variables, selects the action, and authorizes exactly once:
Everything after the gate remains unchanged:
- a caller still cannot enable or disable its own account;
globalIAMSys.SetUserStatusvalidates and persists the requested status;- site replication records the same status and timestamp;
- response and audit behavior use the existing path.
The selector depends only on the requested target state. It does not load the current user, infer a transition from stored state, or make authorization depend on whether the target exists. This keeps authorization deterministic and avoids a read-before-authentication dependency.
Why the repair is safe
The correctness argument consists of five invariants:
- Every valid status maps to exactly one Admin Action.
validateAdminReqis invoked once, so a failed authorization cannot be followed by mutation.- The mutation call is reachable only after the selected action succeeds.
- Invalid status values preserve the old Enable authorization boundary and are still rejected by the existing status-validation path.
- No storage, replication, wire, or client contract changes; only the permission required to reach the existing mutation changes.
The change is a deliberate authorization tightening for Enable-only custom policies that used the disable operation. That tightening is the mechanism that makes admin:DisableUser a real independent capability.
Test design
Pure action mapping
The unit test fixes three selector cases:
| Input | Expected action |
|---|---|
enabled |
EnableUser |
disabled |
DisableUser |
| invalid | legacy EnableUser default |
Four-way IAM authorization matrix
The integration test creates separate users and policies, then exercises the real Admin API:
- a Disable-only client successfully disables a target;
- the same client receives
AccessDeniedwhen enabling it; - an Enable-only client successfully enables the target;
- the same client receives
AccessDeniedwhen disabling it.
Positive assertions alone would not prove least privilege: both policies could accidentally authorize both states and still pass. The two negative cross-action assertions are the security regression tests.
The test removes every temporary user and policy after execution. It runs inside the existing IAM server suite, so it covers request signing, policy attachment, handler authorization, persistence, and Admin-client error decoding rather than testing only the helper.
Repair and verification record
The server checkout originally contained unrelated dependency, generated-credit, checksum-test, and security-document changes, while local main was behind the remote. The two user-status files were isolated into a clean worktree based on current origin/main; no unrelated file entered the repair commit.
Local verification passed:
The signed-off commit 58735ee38 was pushed in PR #73. Its eight remote checks all passed:
- DCO sign-off;
- format, build, and vet;
- lint and generated files;
cmd/tests;internal/tests;- race detector and S3 Select;
- cross compilation;
- vulnerability analysis.
The PR was merged with the repository’s normal merge strategy as 2e2377d1c. Local main was then fast-forwarded only after the two original working files were byte-for-byte and patch-ID identical to the merged result. The unrelated local changes remained intact, and the temporary worktree and task branch were removed after the code became recoverable from main and PR #73.
Least-privilege policy examples
Disable-only operator
This principal can inspect and disable another user, but cannot enable it.
Enable-only operator
This principal can inspect and enable another user, but cannot disable it. Grant both actions explicitly to roles responsible for the complete account lifecycle.
Group-status follow-up
The group endpoint has the same shape as the user endpoint:
It also publishes two existing actions, admin:EnableGroup and admin:DisableGroup. The inherited handler nevertheless authorized every request with EnableGroup before reading status. This was not merely a dead permission: it reversed least privilege in both directions. The wrong principal could disable a group, and the intended disable-only principal could not.
The follow-up adds setGroupStatusAdminAction, deliberately matching setUserStatusAdminAction:
The integration test creates separate EnableGroup-only and DisableGroup-only administrators and a real target group. It proves:
- DisableGroup-only can disable;
- DisableGroup-only cannot enable;
- EnableGroup-only can enable;
- EnableGroup-only cannot disable.
The suite exercises signed Admin requests, policy attachment, handler authorization, IAM mutation, response decoding, and cleanup. Invalid status still selects the legacy Enable action before the existing validation error, so the change does not expose a new pre-authentication oracle. The successful site-replication hook remains after mutation and is not called for denied requests.
This follow-up changes no user behavior and introduces no new policy action. It makes the two already documented group actions enforce the same state-specific contract as their user counterparts.
Compatibility and migration
No client or API migration is required. The endpoint, query parameters, status strings, success response, and Admin-client method are unchanged.
Policy review is required for restricted administrative roles:
- a role that should only disable users needs
admin:DisableUser; - a role that should only enable users needs
admin:EnableUser; - a role that must do both needs both actions;
consoleAdminand otheradmin:*policies are unaffected;- a legacy custom policy containing only
admin:EnableUsercan no longer use that permission to disable users and must addadmin:DisableUserif both operations are intended.
The equivalent rules now apply to group-management roles:
- a role that should only disable groups needs
admin:DisableGroup; - a role that should only enable groups needs
admin:EnableGroup; - a role that must do both needs both actions;
- a legacy EnableGroup-only role can no longer disable groups.
This is a source-level compatibility change in authorization behavior, not a wire-protocol break.
Upstream disposition
As of this record, upstream issue #21478 and PR #21482 are still displayed as open. The upstream repository is archived and read-only. An attempt to leave the single-authorization analysis on the PR was rejected by GitHub because archived, locked discussions cannot accept comments.
The upstream artifacts remain useful provenance but are no longer an actionable delivery path. SILO owns its implemented semantics, tests, merge, release note, and eventual production verification.
Delivery state
| Gate | User repair | Group follow-up on 2026-08-28 |
|---|---|---|
| Design decision | complete | complete |
| Implementation and local tests | complete | complete |
| Independent adversarial review | complete | complete, GO |
| Signed-off commit | complete | 229fe2b3c on main |
| Push, PR CI, and merge | complete | merged 2026-08-29 |
| Tagged SILO release | Server 20260903 | Server 20260903 |
| Release package or container image | See the 20260903 release record | See the 20260903 release record |
| Deployment | not established | not established |
| Production behavior | not established | not established |
| Upstream merge | unavailable; repository archived | not applicable |
Conclusion
The repairs make the authorization model tell the truth. Enabling and disabling users or groups are opposite state transitions with different operational risk, and SILO already exposes different policy actions for each direction. Each handler must therefore select the action from the requested target state and authorize once before mutation.
The code change is small because the design boundary is clear. The durable result is larger: an explicit permission matrix, rejected compatibility alternatives, an invalid-input rule, a four-way integration test, a clean merge record, migration guidance, and an honest release boundary.
The enable/disable repair is separate from the later admin:ChangeMyPassword split. Before upgrading the September candidate, preserve the paired Deny needed for an existing self-password restriction using the password migration guide.
4.19 - Config Environment Files Are Not Shell Scripts
Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.
This record defines the startup contract for MINIO_CONFIG_ENV_FILE and explains the compatibility repair committed in SILO as 2aea7fe9c.
Status on 2026-08-28: implementation, focused tests, the complete
cmdandinternalsuites, tagged tests, race tests, vet, lint, generated-file checks, rebrand guards, build, and an independent local Fable Max review are complete. The commit was merged intomainon 2026-08-29 as2aea7fe9c; tag, package, image, deployment, and production verification remain separate gates.
Scope: environment-file parsing and named-target discovery only. No configuration key, subsystem, value, precedence, storage format, or client API changes.
Compatibility rule: the file is a SILO input format. Supporting an optionalexportprefix does not make it a POSIX shell program.
Too Long; Didn’t Read (TL;DR)
SILO can load startup variables from a file:
The parser accepts assignments such as:
The last two names are important. Multi-target configuration appends the target name verbatim after an underscore. The configuration subsystem does not restrict a target to a shell identifier; names containing -, ., :, digits, or printable Unicode can be discovered and resolved exactly.
A hardening change accidentally validated every key as [A-Za-z_][A-Za-z0-9_]*. It made my-hook invalid and stopped the server during restart even though the previous loader and the configuration target model accepted it. The repair validates what SILO actually needs instead:
- the name is non-empty, valid UTF-8, and made of visible non-whitespace characters;
=and NUL are not allowed in a name;- NUL is not allowed in a value;
- invalid input reports file and line without reporting the value;
- the complete file is parsed before any assignment is applied.
Why the regression was real
The environment-file loader calls os.Setenv after parsing. An operating-system environment is a list of strings, not a shell variable namespace. Shell assignment syntax is narrower because the shell must tokenize and expand variable names in its own language.
Named SILO configuration targets are built differently:
For example:
Target discovery lists variables by the fixed parameter prefix and treats the remaining suffix as the target name. Target lookup reconstructs the same name without uppercasing or sanitizing that suffix. Rejecting - in the file parser therefore broke a valid discover-to-resolve path; it did not protect a shell evaluation path because no shell evaluates the file.
The failure is operationally sharp. MINIO_CONFIG_ENV_FILE is loaded only at startup. A server can continue running with an old process environment, then fail on its next restart after the file or binary changes. Startup must fail on malformed input, but it must not invent a narrower target grammar than the configuration system.
The file grammar
Lines and comments
- blank lines are ignored;
- a line whose first non-whitespace character is
#is ignored; - an optional standalone
exportfollowed by whitespace is removed; exportFOO=valueremains the keyexportFOO; it is not mistaken for the prefix;- the first
=separates key and value, so additional=characters remain part of the value.
The file is not a shell. It does not perform variable expansion, command substitution, backslash processing, or inline-comment interpretation.
Keys
Surrounding whitespace around the key is removed. The remaining key must:
- be non-empty valid UTF-8;
- contain only Unicode graphic characters;
- contain no whitespace,
=, NUL, control, or invisible format characters.
This preserves OS-compatible names and multi-target suffixes while rejecting visually empty or structurally ambiguous keys. A key beginning with a digit or punctuation is accepted by the parser; SILO still reads only the exact names used by its configuration and runtime components.
Values and quoting
Unquoted values are trimmed. To retain leading or trailing spaces, quote the complete value with matching single or double quotes:
The parser removes one matching outer quote pair. It does not interpret escapes inside the quoted value. NUL is always rejected because it cannot be represented in an environment entry.
Failure and secrecy contract
Syntax errors stop startup. Diagnostics include the file path, line number, and the invalid key or error class, but never the value. A password on a malformed line must not be copied into logs.
Parsing is all-or-nothing: a syntax error returns no entries, and assignment starts only after the complete file has parsed. If the operating system rejects a validated assignment, SILO also stops startup and identifies the key and file. Since the process exits, it never serves requests with a partially loaded environment.
The file itself remains a privileged secret-bearing input. Operators must protect it with appropriate ownership and mode; parser validation is not a substitute for filesystem permissions.
Regression matrix
The committed tests cover:
- spaces and tabs around
=; - quoted values with significant spaces;
- standalone
export, including Unicode whitespace after it; - keys beginning with
_, a digit, or punctuation; - named targets using
-,.,:, and Unicode; - exact named-target discovery through the configuration subsystem;
- empty keys, whitespace, NUL, and invisible format characters;
- NUL values;
- multiple
=characters in URLs and tokens; - file-and-line diagnostics that redact values;
- all-or-nothing parse results.
The implementation passed the complete local server verification matrix and a read-only adversarial review. Windows-specific os.Setenv behavior has not been exercised on a Windows runner; unsupported platform rejection remains fail-fast rather than silent.
Compatibility and delivery
No configuration migration is required. Existing ordinary environment names behave unchanged. Files using shell-style whitespace become more predictable, and previously accepted named targets work again.
The visible compatibility changes are intentional:
- invalid or invisible names now fail instead of being silently ignored;
- unquoted surrounding value whitespace is trimmed; quote it when significant;
- malformed input stops startup with a redacted location-aware error;
- a valid punctuation-bearing target is no longer rejected merely because a shell could not assign it with
NAME=valuesyntax.
The parser contract is included in Server 20260903. Verify the artifact running in each deployment separately.
Conclusion
Configuration compatibility depends on validating the format SILO actually consumes. MINIO_CONFIG_ENV_FILE borrows a small amount of dotenv-like syntax for operator convenience, but it is not executed by a shell. The repair restores named-target compatibility while retaining strict NUL, invisibility, redaction, and fail-fast guarantees.
The original #65 also exposed a restart trap: the old parser stored KEY = new as a trailing-space key KEY . A systemd cold start could work because systemd parses EnvironmentFile itself; an admin API re-exec inherited KEY=old, and the malformed new key did not replace it. Restart could succeed with stale credentials or KMS/IdP settings. The repair makes whitespace assignments take effect, so review their intended values before upgrading.
4.20 - Two SSE-C Keys, One CopyObject Response
Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.
This record explains the CopyObject SSE-C checksum response repair committed in SILO as e73436c99.
Status on 2026-08-28: implementation, encryption and key-rotation tests, complete server suites, race tests, static checks, build, and independent Fable Max acceptance review are complete. The commit was merged into
mainon 2026-08-29 ase73436c99; tag, package, image, deployment, and production verification remain separate gates.
Scope: the successful CopyObject XML and HTTP response after the destination object has committed. Stored object bytes, checksum metadata, encryption format, source decryption, federation, replication, and historical objects are unchanged.
Security property: source SSE-C headers may decrypt only source state; destination SSE-C headers may decrypt only committed destination state.
Too Long; Didn’t Read (TL;DR)
An SSE-C copy can use two independent keys:
| Role | Request headers | Purpose |
|---|---|---|
| source | X-Amz-Copy-Source-Server-Side-Encryption-Customer-* |
decrypt the source object |
| destination | X-Amz-Server-Side-Encryption-Customer-* |
encrypt and later interpret the committed destination object |
SILO correctly wrote the destination with its destination key. However, after commit, both the XML generator and the generic PUT-response header helper received the complete CopyObject request. The checksum metadata decrypter intentionally prefers copy-source SSE-C headers when they are present. That priority is correct while reading the source, but wrong when interpreting the committed destination.
With source key A and destination key B:
The object and stored checksum were correct; only the successful response was incomplete. The repair constructs a destination response-header view by removing exactly the three copy-source SSE-C customer headers. It decrypts the destination checksum once, then reuses the resulting map for both XML and HTTP response headers.
Observable failure
The failure requires a checksum-bearing destination and distinct source/destination SSE-C contexts. A representative request supplies:
Before the repair:
- CopyObject returned HTTP 200;
- reading the destination with key B returned the correct body;
- stored destination checksum metadata decrypted with key B and matched the logical bytes;
- the CopyObject XML and HTTP response omitted CRC32 and
ChecksumType.
This is a response-contract defect, not evidence of corrupted object data.
The same ambiguity affects same-object SSE-C key rotation. After metadata has been resealed under key B, the request still carries source key A in the copy-source headers. Response generation must describe the post-rotation object, so it must use B.
Why the global decrypter must not change
The metadata decrypter’s copy-source priority is not itself a bug. Earlier in CopyObject, the server examines source checksum metadata to decide whether to preserve its algorithm, recompute a full-object value, or add the default CRC64NVME checksum. For an SSE-C source, that metadata is protected by the source object key and therefore requires the copy-source headers.
Changing the global priority to prefer destination SSE-C headers would fix the final response while breaking source checksum interpretation. The safe boundary is temporal and object-specific:
The repair applies only at that post-commit boundary.
Selected implementation
Destination response view
The handler clones the request headers and removes exactly:
X-Amz-Copy-Source-Server-Side-Encryption-Customer-Algorithm;X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key;X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key-MD5.
Regular destination SSE-C headers remain. SSE-S3 and SSE-KMS destination metadata needs no customer key and continues through the existing path.
Decrypt once, project twice
Before the repair, CopyObject called decryptChecksums once while building XML and again while writing success headers. For SSE-S3 or SSE-KMS this could repeat KMS unseal work.
The repaired flow is:
The generic setPutObjHeaders wrapper remains available to PutObject, CompleteMultipartUpload, and DeleteObject. CopyObject calls a narrow helper that accepts the already decrypted checksum map. ETag, VersionID, delete-marker, lifecycle prediction, and checksum header behavior remain in one shared implementation.
Regression matrix
The tests cover:
- plaintext source to SSE-C destination;
- compressed and uncompressed SSE-C destinations;
- SSE-C source key A to destination key B;
- checksum value and type in both CopyObject XML and HTTP headers;
- stored checksum decrypted with destination key B;
- destination body readable with B;
- same-object key rotation from A to B;
- checksum response after rotation;
- SSE-S3 source and destination combinations;
- all object-layer backends used by the API test harness.
The final combined tree passed focused encryption tests, the complete cmd and internal suites, the project’s tagged test configuration, full go test -race ./..., vet, lint, generated-file checks, rebrand guards, and a local build. A mirror Fable Max review reported no P0–P2 findings and independently confirmed that source decryption still receives the full request while destination response decryption receives the filtered view.
Compatibility and operational impact
- Successful CopyObject responses: checksum fields that were previously missing now appear when the committed destination has a checksum.
- Stored objects: no rewrite, migration, metadata-format, or encryption-format change.
- Existing objects: unaffected; the defect existed only in the one-time successful response.
- Clients: no request change. Clients already providing both source and destination SSE-C keys receive a more complete S3-compatible result.
- Performance: one metadata checksum decryption instead of two; no additional object read or hash pass.
- Rolling upgrade: old nodes may omit the fields while new nodes return them. Stored objects remain mutually readable.
- Rollback: restores response omission but does not damage objects created while the repair was present.
- Security: no key or digest value is added to logs or error responses. The response carries only the checksum already authorized for the successful write.
This repair does not resolve the separately deferred legacy federation CopyObject branch and does not audit or modify historical compressed-object checksums. Those questions have different data and operational boundaries.
Conclusion
CopyObject is one request with two object identities. Reusing the full request after commit erased that distinction: a source key was allowed to shadow the destination key while describing destination metadata. The durable repair is not a new encryption scheme; it is an explicit context boundary, followed by one decryption and two faithful response projections.
4.21 - Why CompleteMultipartUpload Must Return ChecksumType: Review of PR #57
PR #57, contributed by Shooks (@Dansyuqri), fixed #47. The repair is included in Server 20260903. This records the response contract; production deployment remains specific to the artifact and installation an operator actually runs.
The defect and its scope
Investigation of @cbornet’s #31
separated two defects. Multipart CRC32 completion itself was repaired by
c8590413f and 3e14733f1. After successful completion, the object retained its
checksum type and HEAD could return it, but completion XML omitted that type.
SDK callers therefore saw a missing ChecksumType beside a valid checksum value.
This omission did not corrupt stored data. It made a full-object checksum and a composite checksum harder to distinguish. Their Base64 encodings do not tell a consumer which calculation to reproduce.
Response contract
| Stored state | Completion response |
|---|---|
| Full-object checksum | ChecksumType=FULL_OBJECT with its algorithm value |
| Composite multipart checksum | ChecksumType=COMPOSITE with its algorithm value |
| No additional checksum | No ChecksumType element; no invented checksum |
The S3 completion API defines the two type values. ETag is a separate field and is not a substitute for the additional S3 checksum.
Implementation and evidence
The production change adds ChecksumType string with
xml:"ChecksumType,omitempty" to CompleteMultipartUploadResponse and assigns
cs[xhttp.AmzChecksumType] after decoding the stored checksum metadata. The
generator reuses the same state as the other checksum APIs; it does not
recalculate content or infer a type from part count.
The merged PR
contains response tests for full-object, composite and absent checksums. The
original change also registered the exported field in the then-current rebrand
inventory. That inventory’s exported-symbol section was subsequently removed by
bc3b35f97; it is not a current public API compatibility guarantee.
Compatibility and adjacent work
Readers that ignore unknown XML elements remain compatible. No new algorithm,
stored metadata format, object migration or checksum bypass was introduced.
The UploadPart repair and
completion validation
are separate changes, also included in Server 20260903. In particular,
CRC64NVME + COMPOSITE is rejected, not silently normalized.
4.22 - When the Total Is Unknown: Folder Download Progress
Historical scope: this is the Console 2.2.0 progress repair, included in Server 20260903’s embedded Console. Console 2.4.1 subsequently added file streaming and native browser handoff, eliminating the complete JavaScript ZIP Blob. Transport descriptions and follow-ups in the original PRD below refer to the 2.2.0 design point.
Status: Shipped in SILO Console 2.2.0 (
16960f7ab); the server embeds it since its Console pin was updated (edc8be6ed) · Priority: P1 · Owner:pgsty/silo-console· Related issue:pgsty/silo#62· PRD review: Claude Fable 5 (xhigh) — APPROVE · Implementation review: Claude Fable 5 (xhigh), 2026-08-23 — APPROVE, no P0/P1/P2 findings
SILO Console shows NaN% in Downloads / Uploads while downloading a folder. The ZIP normally keeps streaming and the stored objects are intact, but the progress bar has crossed from “unknown” into an invalid determinate state. Users see a full-looking bar, assume the transfer failed or finished, and retry it.
The proposed repair is intentionally narrow:
A download may enter determinate mode only when it has a finite, positive total measured in bytes applicable to that response. Without such a total, it remains indeterminate until completion, failure, or cancellation.
The server keeps streaming ZIPs. Ordinary files keep their percentages. The frontend gains one safe calculation boundary, reuses its existing indeterminate renderer, and closes one missing cancellation transition. This record defines why that is both sufficient and the smallest truthful fix.
The observed failure
The defect was observed in the then-current silo-console v2.1.1, which is embedded by Silo RELEASE.2026-08-06T00-00-00Z.
Reproduction:
- Put several objects below a prefix such as
folder/. - Stay in the parent listing, select
folder/, and click Download. - Open Downloads / Uploads before the transfer finishes.
- The row displays
NaN%; the ZIP request continues.
The runtime check used a prefix containing about 88.7 MiB and throttled Chromium to preserve the observation window. Two independent downloads produced the same NaN% state.
This is a frontend correctness bug. It is not evidence of corrupted objects, an altered disk format, or a failed S3 GET.
What is actually happening
The visible NaN% is the end of a contract mismatch across three layers.
A prefix has no object size
S3 folders are common prefixes, not stored directory objects. In the listing model, a prefix ends in / and carries size=0. The Console already renders that size as -, correctly treating it as not applicable.
The generated API model marks size as omitempty, so logical zeroes are absent from listing JSON. The single-selection thunk nevertheless passes object.size straight into the download helper: a prefix or zero-byte object therefore supplies undefined at runtime (while synthetic prefix records may supply 0). Neither value is a valid denominator.
A streamed ZIP has no known wire length
The server recognizes the trailing /, recursively lists the objects, then connects a zip.Writer to an io.Pipe. Objects are read, deflated, and copied to the HTTP response as the archive is produced.
That behavior is desirable: the server can send the first bytes without holding the complete archive in memory or on disk. Its consequence is equally deliberate: the final compressed byte length does not exist when headers are sent, so the response has Content-Type: application/zip and a filename, but no Content-Length.
The sum of source object sizes is not a substitute. Source sizes are uncompressed bytes; ProgressEvent.loaded counts response bytes after ZIP compression and framing. They are different units.
A progress event does not imply a computable percentage
The client currently computes every event as:
For a prefix, the denominator is zero or absent. Depending on the value and event, JavaScript produces NaN (loaded / undefined or 0 / 0) or Infinity (positive bytes divided by zero).
The progress callback then writes that non-finite value into Redux and sets waitingForFile=false. That second operation is the decisive state error: the task leaves the existing indeterminate branch merely because an event arrived, not because the event contained a usable total. The determinate progress component receives the invalid value and renders an invalid label.
The complete chain is:
Ordinary non-empty files avoid the defect because the server can stat the object, sets Content-Length, and the list size is positive. If the browser emits a progress event for an empty response, a zero-byte file reaches the same arithmetic boundary as a prefix even though it is a real object; it therefore belongs in the regression contract.
Product contract
The UI needs one honest distinction:
- Determinate means both transferred bytes and total bytes are known in the same unit.
- Indeterminate means the request is active but the total is unknown.
This yields four load-bearing invariants:
These invariants are more general than objectPath.endsWith("/"): they cover prefixes, zero-byte files, malformed metadata, and any future unknown-length response without inventing object-type exceptions.
Goals and non-goals
Goals
- A folder download never displays
NaN%,Infinity%, or a fabricated percentage. - Unknown-length transfers use the existing indeterminate animation.
- Known-length ordinary files retain their current percentage behavior.
- Completion, failure, and cancellation always leave indeterminate mode.
- A zero-byte file never produces a non-finite percentage and still reaches success.
- No non-finite or out-of-range download percentage enters Redux.
- The fix can ship in Console first and then be consumed by Silo as a dependency update.
Non-goals
- Do not pre-generate or buffer a complete ZIP on the server.
- Do not use the sum of uncompressed object sizes as network progress.
- Do not redesign the entire Object Manager state model.
- Do not route folders through the current immediately-completing
BrowserDownloadpath. - Do not solve the browser memory cost of
XMLHttpRequest.responseType="blob"here. - Do not change whether a cancelled row remains visible until the user clears it.
- Do not redesign mid-stream ZIP error signaling after HTTP headers have been sent.
- Do not modify the S3 API, Console API, object layout, or archive contents.
Those are legitimate follow-ups, but coupling them to this defect would enlarge risk without being necessary to restore truthful progress.
The decision
The minimum production repair has four parts.
D1. Calculate only from a valid total
Add a small pure function, separate from DOM and Redux side effects:
The source priority preserves compatibility:
- A finite positive
objectSizeretains the current ordinary-file calculation. - If object size is unavailable but the browser declares the response length computable and supplies a finite positive
event.total, use it. - Otherwise return
null: no truthful percentage exists yet.
The helper’s output contract is complete: either null, or a finite number in [0,100].
D2. Keep unknown totals indeterminate
Change the XHR handler to dispatch only a real percentage:
Download rows already start with waitingForFile=true, and ObjectHandled already renders that state with variant="indeterminate". There is no need to widen Redux to number | null, add another boolean, or change MDS.
When the first valid percentage arrives, the existing updateProgress action stores it and sets waitingForFile=false. When no valid percentage ever arrives, the row remains indeterminate until a terminal action.
D3. Make cancellation terminal
Completion and failure already clear waitingForFile. Cancellation does not. Add the missing transition in cancelObjectInList:
Without that line, the repaired prefix download would remain in the indeterminate rendering branch after abort, masking the Cancelled state. The row continues to follow the current product behavior: it remains as a cancelled record and can be removed manually. Automatic removal is not part of this change.
There is one event-order guard at the XHR boundary as well. abort() first produces readystatechange(DONE, status=0) and only then the abort event; without a status-zero return, the generic DONE branch marks the request failed before onabort can mark it cancelled. DONE/status zero is therefore left to the dedicated onerror or onabort handler, and onabort removes the stored request reference.
D4. Normalize an omitted zero-byte size
The single-selection thunk passes object.size || 0, matching the other download entry point. This restores the API model’s omitted logical zero before the helper checks Blob.size === fileSize, so an HTTP 200 zero-byte object completes at 100% instead of being reported as incomplete.
D5. Keep the server stream unchanged
The folder handler continues to generate a deflated ZIP through io.Pipe and omit Content-Length. No API, archive, storage, or resource-management contract changes.
State machine
| State | waitingForFile |
percentage |
Terminal flag | Rendering |
|---|---|---|---|---|
| Queued / no valid progress yet | true |
0 |
none | indeterminate |
| Unknown-total transfer | true |
0 |
none | indeterminate |
| Known-total transfer | false |
0..100 |
none | determinate percentage |
| Completed | false |
100 |
done=true |
success |
| Failed | false |
last value | failed=true, done=true |
error |
| Cancelled | false |
0 |
cancelled=true, done=true |
cancelled |
The state does not move back from determinate to indeterminate. If a later event lacks a valid total after a valid percentage was observed, the handler simply retains the last valid value.
Failed and Cancelled both set done=true in the existing reducers. ObjectHandled uses done to change its close button from “abort request” to “remove record”; this repair preserves that behavior. The cancelled Redux value remains 0, while the existing ProgressBarWrapper renders a full orange terminal bar with a Cancelled label because ready=true. That established presentation is not part of this repair.
waitingForFile is not the ideal long-term name for “no computable progress.” Renaming it or replacing the booleans with a discriminated union would improve the model, but that is a separate refactor. In this repair, the field already expresses and renders the required state, so reusing it minimizes compatibility risk.
Why this is sufficient
The repair closes the bug by cases.
Ordinary non-empty file
objectSize > 0, so the helper uses the same denominator as today. The result is finite and clamped, updateProgress enters determinate mode, and completion still sets 100%.
Current streamed folder
objectSize is normalized to 0, while lengthComputable=false and event.total=0. The helper returns null; no invalid action is dispatched, so the row remains indeterminate. Completion sets waitingForFile=false, percentage=100, and done=true.
Future response with a real length
If a proxy or later server implementation provides a trustworthy response total, lengthComputable=true and event.total>0. The same code automatically produces a real percentage without another product change.
Zero-byte file
The omitted listing size is normalized to zero, and both totals are then zero, so an intermediate percentage is mathematically undefined. The row stays indeterminate for its usually brief lifetime; the zero-byte Blob now equals the normalized expected size, and the successful response transitions directly to 100%. 0/0 is never evaluated.
Failure and cancellation
Failure already exits indeterminate. The added cancellation transition does the same on abort. No terminal row can continue to look active merely because its total was unknown.
Mathematically, division occurs only when total belongs to (0, +infinity). The result is then clamped to [0,100]. Therefore neither NaN nor Infinity can cross the calculation boundary into Redux or the determinate renderer.
Rejected alternatives
Buffer the ZIP to obtain Content-Length
The server could generate the complete archive in memory or a temporary file, measure it, and then send it. That would provide an exact wire total, but at the cost of memory or disk pressure, delayed first byte, cleanup complexity, and worse concurrent-download behavior. An observability defect does not justify discarding streaming.
Sum the objects under the prefix
That sum is uncompressed logical data. event.loaded measures compressed response bytes plus ZIP framing. The units differ, so the bar could stop below 100%, exceed 100%, or move according to compression ratio rather than transfer completion. Reject.
Convert invalid progress to 0%
This hides the string but lies about the state: determinate 0% means the total is known and no portion has transferred. Users would still interpret the transfer as stalled. Unknown must remain unknown.
Special-case paths ending in /
That fixes the reported prefix but misses a real zero-byte object, invalid metadata, and other unknown-length responses. The correct boundary is denominator capability, not object type.
Send folders through BrowserDownload
The current large-file path creates an anchor and immediately calls the completion callback after clicking it. It cannot report true completion, console-managed cancellation, or a subsequent HTTP failure. It may be the basis of a later streaming-download design, but today it would replace one lie with another.
Sanitize inside ProgressBar
A generic component guard could be useful defense in depth, but it would leave invalid data in Redux and hide the broken state transition from every other consumer. The primary repair belongs where progress becomes application state.
Introduce percentage: number | null now
A discriminated progress state would be cleaner than the current booleans if the Object Manager were being redesigned. Adding null while retaining waitingForFile, done, failed, and cancelled would instead create more contradictory combinations. Removing the old fields is larger than this bug requires. Reuse the already-rendered indeterminate state now; redesign it separately.
Requirements and acceptance
Functional requirements
- FR1: An unknown total keeps the task indeterminate.
- FR2: A finite positive object size preserves ordinary-file percentages.
- FR3: A finite positive
event.totalis a fallback only whenlengthComputable=true. - FR4: Every dispatched percentage is finite and within
[0,100]. - FR5: A zero-byte file never displays non-finite progress and reaches success.
- FR6: Completion, failure, and cancellation leave indeterminate mode.
- FR7: Versioned objects, anonymous downloads, previews, and long-filename entry points retain their existing call contract.
Non-functional requirements
- No new server CPU, memory, disk-buffer, or request cost.
- No new frontend dependency or build step.
- No change to the S3 API, Console API, ZIP content, or stored objects.
- The calculation must be testable without a DOM or live store.
- TypeScript typecheck and the production frontend build must pass.
Acceptance criteria
- While a folder ZIP without
Content-Lengthis active, its row shows an indeterminate animation and no percentage text. - On successful completion, the row reports success/100% and the ZIP can be opened.
- A normal non-empty file continues to show finite determinate progress and completes at 100%.
- A zero-byte file never shows
NaN%orInfinity%and completes successfully. - Cancelling an unknown-total download aborts the request and shows Cancelled, not an active animation.
- No download path can place a non-finite or out-of-range percentage in Redux.
Test plan
Pure calculation matrix
Use the existing @playwright/test runner for the pure module rather than adding a test framework. This needs one config-only addition in web-app/playwright.config.ts: a dependency-free unit project, for example with testMatch: /.*\.unit\.ts/. The existing chromium project depends on the auth setup against a live Console at localhost:9090; pure calculation and reducer tests must not be gated by that environment. No new dependency is introduced.
| Case | loaded |
objectSize |
lengthComputable |
event.total |
Expected |
|---|---|---|---|---|---|
| Ordinary file, halfway | 50 | 100 | false | 0 | 50 |
| Common prefix | 1024 | 0 | false | 0 | null |
| Initial zero over zero | 0 | 0 | false | 0 | null |
| Response-total fallback | 50 | 0 | true | 200 | 25 |
| Zero total is unusable | 0 | 0 | true | 0 | null |
| Loaded exceeds total | 150 | 100 | true | 100 | 100 |
| Invalid object size | 10 | NaN |
false | 0 | null |
| Omitted zero size | 10 | undefined |
false | 0 | null |
| Invalid response total | 10 | 0 | true | Infinity |
null |
| Negative loaded | -1 | 100 | true | 100 | null |
State tests
Cover the transition contract directly:
- A new download starts with
waitingForFile=true. - No valid progress action means it remains indeterminate.
- Valid progress produces a finite value and
waitingForFile=false. - Complete produces
done=true,waitingForFile=false,percentage=100. - Failure produces
failed=true,done=true,waitingForFile=false. - Cancel produces
cancelled=true,done=true,waitingForFile=false,percentage=0.
Browser regression
Use the real Console test instance and Chromium:
- Create a temporary bucket with several objects below
folder/. - Select the prefix from its parent and start the download.
- Apply CDP download throttling so the intermediate state is observable.
Throttled runs must raise the default 30-second test timeout with
test.setTimeout. - Open Downloads / Uploads and verify that the row exists, has no percentage label, and contains neither
NaN%norInfinity%. - Cancel it and verify the Cancelled terminal state.
- Restore network conditions in
finally. - Download again without throttling, wait for the browser download, and verify the ZIP.
- Repeat the relevant assertions for one ordinary non-empty file and one zero-byte file.
- Remove the bucket, objects, downloads, and temporary files in teardown.
The current Playwright project is Chromium-only, so CDP is an acceptable test mechanism. If Firefox or WebKit projects are later enabled, keep the pure and state tests cross-browser and gate only the throttled observation behind the Chromium project.
Implementation boundary
Expected Console changes:
- Add
downloadProgress.tscontaining the pure calculation. - Change
Objects/utils.tsto dispatch only a non-null percentage, let status-zero terminal events reach their dedicated handlers, and clean up an aborted request. - Normalize omitted zero sizes in the single-selection thunk.
- Change
cancelObjectInListto clearwaitingForFile. - Add calculation, state, and browser regression coverage using existing dependencies, with a dependency-free
unitproject inplaywright.config.ts.
Expected unchanged code and contracts:
- The Go folder-download handler and its streaming ZIP.
ObjectHandled,ProgressBarWrapper, and MDS.IFileItem.percentage: numberand the existing thunk callback types.- S3 and Console API routes.
- Stored object and archive formats.
Delivery and rollback
The fix belongs in pgsty/silo-console, not the Silo server repository where the issue was reported.
Delivery order:
- Transfer or cross-reference issue #62 to
pgsty/silo-console. - Implement the bounded Console change.
- Pass typecheck, production build, pure/state tests, and real browser regression.
- Publish a new Console release.
- Update Silo’s pinned Console pseudo-version or release dependency.
- Build a Silo candidate and repeat folder, ordinary-file, zero-byte, cancel, and ZIP-integrity checks.
- Publish Silo and record both affected and fixed versions on the issue.
There is no data migration. If the frontend change regresses, Silo can roll back only the Console dependency; server data and API behavior remain compatible.
Definition of done
- The calculation returns only
nullor a finite[0,100]number. - Active unknown-total folder downloads render indeterminate.
- Ordinary files retain determinate progress.
- Zero-byte files never render invalid progress.
- Complete, failed, and cancelled rows all leave indeterminate mode.
- The streamed ZIP and server response contract remain unchanged.
- Typecheck, production build, and automated regressions pass locally.
- Console 2.2.0 is published.
- Server 20260903 includes the updated Console; see its release evidence.
Follow-up work
Five adjacent improvements deserve separate design records:
- Stream large folder downloads directly to the browser or filesystem instead of holding the full Blob in memory.
- Replace the Object Manager’s boolean combination with a discriminated progress/terminal state.
- Improve end-to-end integrity and error signaling for ZIP failures after headers have been sent.
- Add a generic non-finite-value guard to shared progress components as defense in depth.
- Repair the pre-existing Blob JSON error decoder and request-trace cleanup on HTTP failure paths.
None is required to stop the current UI from lying. The next maintenance iteration should first restore the smallest honest contract: known totals get percentages; unknown totals remain unknown.
Later implementation and the original design
The Console 2.2.0 integration also made size an always-present JSON field and repaired ZIP error propagation: unreadable entries are no longer silently skipped; errors before output can return 500 and errors afterward interrupt the stream. The no-API/resource-management-change statements above describe only the original progress-calculation patch, not the whole release.
Current single-folder downloads are handed to the browser. A completed Console row records the handoff, not completion of all bytes; track and cancel the transfer in the browser download manager. Multi-selection file-writer/native handoff paths are described in Console 2.4.1. Size normalization remains defensive support for old responses, not evidence that the current model omits zero values.
4.23 - A ListObjects Shortcut Must Not Turn a Missing Bucket into an Empty One
Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.
This document records the problem analysis, design discussion, and repair decision for SILO #32 and PR #37.
Status on 2026-08-26: PR #37 was updated to the DCO-signed head
e9c5340be, formally approved, and merged as49c8aeac4; #32 closed automatically. DCO, VulnCheck, and all six Go CI jobs passed on the exact PR head; the post-mergemainVulnCheck and all six Go CI jobs also passed. No tagged release, package, container image, deployment, or production endpoint has yet been verified to contain the repair.
Scope: verify bucket existence only for three listing shortcuts that bypass storage; do not restore the genericcheckBucketExist, change the normal listing path, or introduce an existence cache.
Release boundary: local commit, push, remote CI, merge, tag, package, container image, deployment, and production verification are independent gates.
Too Long; Didn’t Read (TL;DR)
The problem is real and worth fixing. A normal ListObjects, ListObjectsV2, or ListObjectVersions request against a missing bucket reaches storage and receives BucketNotFound. Three inputs, however, return early:
- a marker outside the prefix;
max-keys=0;- a prefix beginning with
/, including thePrefix="/"boto3 reproduction from #32.
Those branches return io.EOF directly. The caller treats EOF as a successful end of listing, so the client receives an empty 200 rather than S3’s 404 NoSuchBucket. The identity of the same missing resource changes from an error to success solely because the selection parameters differ. That breaks S3 compatibility and blocks a real user’s upgrade from the pre-regression release.
The repair must not put an expensive bucket check back in every listing. The selected design replaces only the three bare io.EOF returns with a small helper. The helper calls GetBucketInfo once: it returns the real error if the bucket is absent or cannot be confirmed, and preserves io.EOF when the bucket exists. The normal listing hot path is untouched. Only requests that would otherwise exit before storage pay the extra peer-and-disk fan-out.
That decision has now been executed: the strengthened repair passed local review, the exact PR head passed every remote check, and the expected-head-guarded merge entered a green main.
What is the problem?
One API exposes two bucket-existence semantics
#32 reproduces the defect by calling the following against a missing bucket:
AWS S3 raises NoSuchBucket; SILO returns a successful empty listing. The difference is not in authentication, routing, or XML serialization. It comes from the object-layer listPath control flow:
/ is not the only trigger:
| Shortcut condition | Why the result must be empty | Defect before the repair |
|---|---|---|
| Marker does not begin with the prefix | The implementation does not scan this disjoint range | Returns EOF without confirming the bucket |
max-keys=0 |
The caller asks for zero keys | Incorrectly equates “zero results” with “valid resource” |
Prefix begins with / |
SILO’s flat key space produces no entries for this form | The filter short-circuits before bucket identity |
For an existing bucket, returning an empty listing from these branches is a reasonable optimization. For a missing bucket, the same EOF masks the resource error that should take precedence.
The regression has a known origin
The reporter confirmed correct behavior in RELEASE.2024-01-29T03-56-32Z and the regression beginning with RELEASE.2024-01-31T20-20-33Z. The corresponding upstream change is minio/minio#18917 / 80ca12008. It removed GetBucketInfo from generic argument checks and relied on actual Put, List, and Multipart storage operations to expose a missing bucket.
That optimization works on normal paths but leaves a gap: an early-return path never reaches the storage operation that is now responsible for producing the error. #32 does not require a broad rollback of the upstream optimization. It repairs the overlooked control-flow exits.
Why fix it?
The S3 contract explicitly requires NoSuchBucket
Both AWS ListObjects and ListObjectsV2 define NoSuchBucket as HTTP 404 when the specified bucket does not exist. prefix, marker, start-after, and max-keys select listing results; they must not turn a missing bucket identity into a successful request.
ListObjectVersions shares the same object-layer listing engine. Giving V1, V2, and version listings the same existence behavior on the same shortcut inputs prevents the three public APIs from diverging further.
An empty 200 changes client decisions
An empty 200 and a 404 are not interchangeable presentation details:
- 404 tells provisioning or test code to create the bucket, fix configuration, or stop;
- an empty 200 asserts that the bucket exists but has no matching objects;
- SDKs, synchronization tools, and integration tests continue down different branches;
- a test using SILO as an S3 substitute can pass locally and fail against AWS.
#32 also establishes a direct upgrade impact: an application relying on the older correct behavior cannot upgrade past the regression. The repair restores both S3 parity and upgrade compatibility.
The repair surface is narrow and testable
The bug is confined to three adjacent early returns. It does not involve object data, metadata formats, sorting, pagination-token encoding, permissions, or wire schemas. A very small production change can be pinned down with object-layer and HTTP-level contracts, so the benefit clearly exceeds the implementation risk.
Why not restore the global check?
Upstream did not remove generic GetBucketInfo as incidental cleanup. The motivation for #18917 states that checking the bucket before every Put, List, and Multipart operation fans out across servers; even after vectorization, the cost becomes visible beyond 100 nodes.
In current SILO, erasureServerPools.GetBucketInfo calls S3PeerSys.GetBucketInfo. That operation concurrently asks every peer and reduces quorum per pool, while each peer checks its local bucket state. It is not a cheap in-memory map lookup.
Two extremes are therefore unacceptable:
- never check: keep the incorrect empty 200;
- check before every List: restore semantics while undoing a critical large-cluster optimization.
The actual design question is whether the check can be confined to branches that never touch storage and therefore cannot discover the missing bucket naturally. It can.
How is it fixed?
Replace only three bare EOF returns
In cmd/metacache-server-pool.go, each shortcut previously executed:
It now executes:
The helper has only two classes of outcome:
- existing bucket: preserve the previous empty-list behavior;
- missing bucket: pass
BucketNotFoundinto the existing error mapping, producing HTTP 404NoSuchBucket; - state cannot be confirmed: propagate quorum, offline, timeout, or context errors instead of fabricating success.
The normal listMerged, metacache scan, sorting, pagination, and response-generation paths do not change.
Why the helper belongs here
The check must sit next to the shortcut for three reasons:
- only this layer knows that it is about to bypass every storage access;
- moving it into generic argument validation charges every call;
- moving it into the scan layer cannot help because these branches never scan.
The name intentionally states the boundary. This is not a new generic checkBucketExist; it restores missing existence semantics immediately before a shortcut returns EOF.
Do not add a cache
A bucket-existence cache could reduce fan-out but immediately creates invalidation questions for create, delete, site replication, recovery, and expiry. Adding a second source of truth for three low-frequency shortcuts costs more complexity and consistency risk than it saves.
The selected implementation uses the existing GetBucketInfo source of truth. If future telemetry shows that large clusters receive frequent max-keys=0, slash-prefix, or disjoint-marker probes, the project can evaluate a dedicated metadata fast path, rate limiting, or a carefully invalidated cache using real data rather than speculative machinery in this compatibility patch.
Test and review evidence
Object-layer contract
The object-layer test runs against single-drive and multi-drive erasure setups and exercises four inputs:
- slash-prefixed prefix;
- zero limit;
- marker outside prefix;
- a regular prefix as a control that still receives the error naturally from storage.
Each case covers ListObjects, ListObjectsV2, and ListObjectVersions, using the typed isErrBucketNotFound predicate rather than brittle English error-string comparison.
HTTP contract
The handler test sends genuine signed requests for all three public APIs:
| API | Request shape | Assertion |
|---|---|---|
| ListObjects | GET /missing-bucket?prefix=/ |
HTTP 404 and XML code NoSuchBucket |
| ListObjectsV2 | Add list-type=2 |
HTTP 404 and XML code NoSuchBucket |
| ListObjectVersions | Add versions |
HTTP 404 and XML code NoSuchBucket |
The HTTP test uses the real slash-prefix reproduction from #32. The other two shortcuts are enumerated at the object layer. This proves final wire behavior without repeating the full matrix in the slower handler fixture.
Local quality gates
The improved local commit passed:
The full local cmd test completed in 116.215 seconds. An independent local Claude Code review used the Fable model at Max effort to inspect the exact tree, call paths, error mapping, tests, performance boundary, and this decision. Its verdict was GO, with no mandatory pre-merge change.
The DCO-signed PR head e9c5340be then passed eight remote checks: DCO, VulnCheck, and six jobs in Go CI. After merge, the resulting main commit 49c8aeac4 independently passed VulnCheck and all six Go CI jobs. The slowest checks were PR cross-compile at 9 minutes 47 seconds and post-merge cross-compile at 9 minutes 30 seconds.
Can it introduce new problems?
Shortcut requests now fan out across the cluster
This is the most important and deliberately accepted cost. A shortcut on an existing bucket used to be little more than a local branch; it now calls GetBucketInfo. Directional local microbenchmarks observed:
| Path | Observed magnitude |
|---|---|
| Shortcut before the repair | about 0.55 μs, 7 allocations |
| Repaired single-drive shortcut | about 7.8–8.1 μs, 45–47 allocations |
| Repaired 32-drive shortcut | about 70–81 μs, 977 allocations |
| Normal 32-drive listing | about 0.95 ms |
These numbers show local relative cost only; they are not a latency prediction for a 100+ node deployment. Real distributed execution adds peer networks, quorum, and slowest-node tail latency, potentially making the gap much larger. That is precisely why the check must not expand into the normal listing path.
The risk concentrates in malformed or probe-style traffic. A misconfigured client polling max-keys=0, a slash prefix, or disjoint markers at high frequency can amplify what was a cheap request into peer-and-disk work. After merge, the actual frequency of these inputs should be observed through S3 traces or metrics; rate limiting or optimization should follow evidence.
A degraded cluster exposes more real errors
Previously, a shortcut could return an empty 200 while peers were offline or bucket quorum was unavailable because it never consulted cluster state. The repair can return quorum, timeout, or service errors in those conditions.
That is more honest behavior, not an availability regression: if the server cannot establish that the bucket exists, it must not assert a valid empty bucket. Clients depending on unconditional empty success will nevertheless observe a behavior change.
Bucket create/delete races are not linearizable
GetBucketInfo and returning the empty result are two actions. The bucket can be deleted immediately after the check, or created immediately after a missing-bucket result is formed. This patch does not and should not add a transaction spanning bucket lifecycle to a listing shortcut.
This is the same concurrency class as other APIs that validate a resource before acting. The repair guarantees that the request no longer succeeds with no existence evidence at all; it does not promise a cross-node, cross-lifecycle linearizable snapshot of an empty listing.
Clients relying on the bug will receive 404
Some clients may have adopted the missing bucket’s empty 200 as fact. They will now enter an error branch. This is a visible compatibility change, but it restores the documented S3 contract and the pre-regression behavior. Preserving the bug merely transfers upgrade cost to clients that correctly rely on 404.
Two adjacent edges remain out of scope
The adversarial review recorded two non-blocking P3 boundaries:
- When resuming a metacache continuation, the
c.fileNotFoundbranch still returns bareio.EOF. A stale or crafted continuation token used after bucket deletion could theoretically receive an empty 200. AddingGetBucketInfothere would affect normal continuation traffic and needs a separate performance and error-precedence design. - Some V1 and version-list marker/prefix combinations return
NotImplementedduring HTTP handler validation before reaching the object layer; the V2start-afterroute can reach it. This patch fixes storage shortcuts masking a missing bucket; it does not redefine precedence between malformed parameters and resource errors.
Neither blocks merge. The first is outside #32’s ordinary initial-list reproduction; the second is inherited handler behavior. Recording them prevents “all three shortcuts are covered” from being overstated as byte-for-byte AWS parity for every possible parameter combination.
Alternatives considered
Keep upstream behavior
This has zero performance change and minimizes fork divergence. It also keeps a documented S3 incompatibility, a regression with a known release boundary, and a misleading result when SILO is used as an integration-test substitute. For a narrow and well-tested compatibility repair, that tradeoff is no longer justified.
Restore generic checkBucketExist
This covers every path at once but reintroduces peer fan-out into every Put, List, and Multipart operation, directly undoing the large-cluster optimization from #18917. The cost is disproportionate and the option is rejected.
Fix only Prefix="/"
That passes the single issue reproduction but leaves the same root defect in max-keys=0 and marker-outside-prefix. The branches are adjacent and share the same semantics, so one helper is simpler and less likely to regress.
Add a bucket-existence cache
This makes shortcuts cheaper but requires semantics for create, delete, replication, recovery, and stale TTL windows. There is no telemetry showing enough shortcut traffic to justify that complexity, so it is not selected.
Complexity and cost-benefit
| Dimension | Assessment | Rationale |
|---|---|---|
| Production-code complexity | Low | Three call sites and a seven-line helper; no new state, dependency, or format |
| Test complexity | Low to medium | V1, V2, versions, three shortcuts, a control, and HTTP mapping all need coverage |
| Normal-path risk | Very low | No check is added to the listMerged hot path |
| Shortcut runtime cost | Materially higher | A local EOF becomes cluster-wide GetBucketInfo |
| Compatibility value | High | Restores 404 NoSuchBucket, pre-regression behavior, and S3 test fidelity |
| Operational complexity | Low | No migration, configuration, feature flag, cache, or cross-repository dependency |
The overall cost-benefit is favorable. The reason is not that GetBucketInfo is cheap—it is not—but that its cost is strictly limited to three shortcuts that otherwise cannot discover the missing bucket. A narrow performance cost in exchange for explicit protocol correctness is better than either a global rollback or indefinitely preserving the incorrect behavior.
Acceptance decision and remaining gates
The final decision was: accept and merge the strengthened PR #37 revision without expanding the production scope.
The accepted sequence was:
- replace the old fork head with the current-
main, DCO-signed revision while preserving Jason Lin as a co-author; - retain typed error predicates, V1/V2/version-list object-layer coverage, and HTTP-level 404 /
NoSuchBucketassertions; - update the PR description with the shortcut fan-out cost and unchanged normal-path boundary;
- approve the fork workflows and require all eight reported checks to pass on exact head
e9c5340be; - submit a formal approving review against that head;
- merge with an expected-head guard, producing
49c8aeac4, automatically close #32, and require the resultingmainGo CI and VulnCheck to pass independently.
No cache, feature flag, additional abstraction, or continuation-token redesign was required. High-frequency shortcut traffic and large-cluster tail latency remain observability follow-ups, not reasons for speculative code expansion.
Repository integration is complete. A tag, package, docker.io/pgsty/silo image, deployment, and real S3-client verification must still complete before the repair can be described as delivered to users.
Conclusion
The issue is not merely “a slash prefix reports the wrong error.” The listing engine uses io.EOF to mean two different things: an empty result from an existing bucket and an early exit that never established whether the bucket exists. Removing generic existence checks for large-cluster performance was a sound upstream optimization, but the shortcuts violate its premise that a real storage operation will naturally surface a missing bucket.
The selected repair restores that premise by calling the existing GetBucketInfo only at three storage-bypassing exits. It makes those requests more expensive and exposes real errors on degraded clusters; both are explicit costs. In return, SILO restores S3’s 404 semantics, upgrade compatibility, and test fidelity while preserving the upstream optimization on the normal listing hot path.
This worthwhile, controlled compatibility fix is now merged and green on main; the repair is included in Server 20260903; production deployment remains installation-specific.
4.24 - Read-Only Checksum Audit and Reliable CLI Output
This is the design and implementation record for MCLI’s read-only checksum verification workflow and pgsty/mc#5, the non-TTY output defect found during release review.
Status: shipped in the final mcli 20260903 release. The command merged to
mainthrough pull requests #8 and #13, is exercised against a real SILO server in hosted CI, and pgsty/mc#5 is closed. Server 20260903 images bundle mcli 20260903, including this command.
Owner:pgsty/mc.
Tracking: pgsty/mc#5.
Safety boundary: verification is read-only; repair is not part of this command.
Too Long; Didn’t Read (TL;DR)
Historical CopyObject implementations could calculate a stored additional
checksum over transformed storage bytes instead of the logical bytes returned
by S3. mcli checksum verify inventories objects and independently streams the
logical body through the recorded algorithm. Each candidate becomes MATCH,
MISMATCH, NO_CHECKSUM, WOULD_VERIFY (dry run), one of ten UNKNOWN_*
classifications, or one of three SKIPPED_* results.
The first implementation worked in a terminal but printed nothing when stdout was redirected. MCLI automatically marked non-TTY execution as quiet to disable progress UI, and the new command accidentally treated that internal state as a user request to suppress audit records. The repair separates semantic output from progress suppression without changing global quiet behavior or enabling progress bars in CI.
Command and scope
Version one supports CRC32, CRC32C, CRC64NVME, SHA1, and SHA256 checksums marked
as FULL_OBJECT. It can select one object, an exact VersionID, current objects
under a prefix, all versions, or exact entries from a JSON Lines manifest. It
also supports SSE-C key mappings, time and size filters, dry-run estimation,
bounded workers, download limits, JSON output, and an optional JSON Lines report.
It does not verify COMPOSITE checksums, infer type from an ETag, inspect
xl.meta, identify the historical writer with certainty, or repair metadata.
The endpoint must report the checksum type (x-amz-checksum-type) alongside
the checksum; on one that does not, every checksummed object is classified
UNKNOWN_CHECKSUM_TYPE rather than guessed at.
Read-only data path
For every selected object, MCLI:
- sends
HEADwith checksum mode enabled and retains every supported checksum plusChecksumType; - rejects unsupported or ambiguous states as
UNKNOWN_*instead of guessing; - streams
GETlogical bytes through bounded hashers without writing the body to disk; - uses VersionID pinning, or
If-Matchplus a secondHEADfor mutable unversioned/null objects; - compares independently calculated values with the stored values.
The S3 boundary allows LIST, HEAD, and GET only. Tests fail if a write method reaches the mock endpoint.
Result and exit contract
Every candidate produces one stable result:
| Result | Meaning |
|---|---|
MATCH |
Every supported stored checksum matches the returned logical bytes |
MISMATCH |
At least one stored checksum differs |
NO_CHECKSUM |
No additional checksum exists; the body is not read |
WOULD_VERIFY |
Dry-run found a supported full-object checksum |
UNKNOWN_* |
MCLI cannot make a reliable statement |
SKIPPED_* |
A filter intentionally excluded the object |
The summary carries objects, a verified count, the count of every result
status, and incomplete. verified is MATCH plus MISMATCH: the only results
that actually streamed a body through a hasher. A run that enumerated many
objects and verified none is visible as such.
--fail-on accepts mismatch, unknown, no-checksum, any, or none. The
default any returns exit 1 for mismatches and incomplete verification.
no-checksum returns exit 1 when any object carries no checksum or when
nothing was verified at all, so an empty prefix or a stale manifest cannot
pass as a clean audit. Dry-run does not apply --fail-on. Argument,
authentication, enumeration, and report-write failures remain command failures
rather than object classifications.
In particular, SKIPPED_TOO_LARGE makes the default any return exit 1 because
the size cap leaves the audit incomplete. Time-filter and delete-marker skips do
not fail by themselves.
Output and automation contract
Object records and the final summary are semantic output:
- Unless the caller explicitly sets
--quiet,-q, orMC_QUIET=true, stdout receives every object record and the final summary in both TTY and non-TTY execution. - Non-TTY
--jsonemits exactly one compact JSON value per line. TTY JSON keeps MCLI’s existing pretty presentation. - Global flags work at the app,
checksum, andverifylevels. --reportis independent of stdout. It still writes object records and the final summary as JSON Lines when explicit quiet suppresses stdout.- Output transport does not change
--fail-ondecisions.
The distinction matters because MCLI’s historical globalQuiet has two inputs:
an explicit quiet flag and an automatic non-TTY state used to disable progress
UI. Changing that global would risk re-enabling progress output across copy,
get, put, mirror, and other commands.
The selected repair is command-local. It walks the full CLI context chain for
explicit quiet/JSON flags because the CLI library’s GlobalBool stops at the
nearest ancestor flag set. It also restores JSON Lines mode inside the checksum
action because nested Before hooks can reset it after an app-level --json.
No other command’s progress or output behavior changes.
Report, secrets, and operational cost
Report files are created with mode 0600, must not already exist, and contain
metadata/results rather than object bodies or SSE-C keys. The manifest likewise
contains only bucket, key, and optional VersionID.
Verification downloads every supported object body. Operators should use
--dry-run, --max-size, time filters, --max-workers, and the global download
limit to bound cost and load. NO_CHECKSUM and UNKNOWN_* counts must remain
visible; neither may be presented as successful verification.
What a mismatch proves
A mismatch proves only that the additional checksum returned at verification time does not describe the logical bytes returned at verification time. It does not prove that a particular historical compression defect created the object, and it is not an external source-of-truth comparison.
Do not overwrite checksum metadata in place. Audit and classify first. For a
confirmed, operationally relevant mismatch, prefer a new key or new version,
verify the replacement, then switch consumers deliberately. Leave UNKNOWN_*
objects out of automatic repair.
Verification record and release boundary
The local acceptance matrix covers TTY human/JSON, non-TTY pipes, regular-file
redirects, app/parent/leaf JSON and quiet flags, environment quiet, report under
quiet, report-write failure, and MISMATCH/UNKNOWN exit status. It also includes
real historical MATCH, MISMATCH, and unsupported-composite objects on a local
S3 server.
The command shipped in the final mcli 20260903 release from a
signed tag at the tip of main, with the functional suite - including a
checksum verification run against a real SILO server - green for that commit,
and pgsty/mc#5 is closed. Server 20260903 images bundle that client. A production audit remains a separate execution with its own evidence.
JSON consumers should check schemaVersion: 1 and distinguish per-object records (type: object) from the final summary (type: summary). --max-workers defaults to 4 and accepts 1–64; it bounds concurrent object work, not total memory or server I/O.
4.25 - Optional Checksums, Mandatory Failure: Repairing UploadPart and UploadPartCopy Compatibility
Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.
This is the complete design and implementation record for SILO #46. The repair was not merely a changed if statement. One apparently optional S3 header reached into multipart completion semantics, copy responses, compression and encryption pipelines, compatibility baselines, and release verification.
Status: merged into
mainas7fea6d5a5on 2026-08-24 (pgsty/silo#46 closed); included in Server 20260903; deployment verification is installation-specific.
Owner:pgsty/silo, the SILO server repository.
Tracking: #46.
Independent follow-ups: #63 CopyObject + compression checksum, #64 federated UploadPartCopy checksum.
Adversarial review: local Claude Code, Fable 5,--effort max; final verdict GO, with no blocking findings.
Too Long; Didn’t Read (TL;DR)
A multipart upload splits a large file into smaller parts. A client may attach a checksum to each part so the server can verify the transfer, but AWS defines that checksum as optional. SILO used to treat it as mandatory: an ordinary UploadPart failed without one, and UploadPartCopy could never work because it has no part-body checksum to provide.
After the repair, SILO still validates a checksum when the client sends one. When the client omits it, SILO computes the checksum while reading the original bytes and saves the result. This happens before compression and encryption, requires no second read, and changes no on-disk format. The result is AWS-compatible behavior without weakening data integrity.
Decision
When a multipart upload declares a checksum algorithm in CreateMultipartUpload, SILO applies this contract:
- If the client supplies a part checksum, the server continues to validate it. A wrong value or algorithm fails and is never hidden by fallback computation.
- If the client omits the part checksum, the server computes it in one pass with the MPU algorithm over the logical plaintext stream, before compression and encryption, and persists the result.
- A normal
UploadPartechoes a checksum response header only when the client supplied the checksum. A server-computed fallback is not echoed. UploadPartCopyhas no client part-body checksum, so the server computes the value and returns it inCopyPartResult.ListPartsreturns the persisted part checksum.FULL_OBJECTcompletion continues to linearize the full checksum from stored part checksums.COMPOSITEcompletion continues to require a checksum for every part; clients can recover those values withListParts.- Computation occurs during the existing read. Completion never re-reads the entire object merely to manufacture missing state.
In one sentence:
The optional input is the client-provided checksum value, not the server’s responsibility to maintain a consistent checksum-enabled MPU.
How we found it
The defect surfaced while investigating a different multipart checksum issue, #31.
#31 concerned CompleteMultipartUpload: for FULL_OBJECT, a client can complete with part numbers, ETags, and an optional full-object checksum without retaining every part checksum in the completion XML. Tracing that path backward exposed a stronger, earlier condition in erasureObjects.PutObjectPart:
Once an MPU declared a checksum algorithm, every UploadPart had to carry the matching x-amz-checksum-* value. Omitting it returned:
API-level probes reproduced the behavior on both the single-drive and erasure backends.
Reviewing CopyObjectPartHandler raised the severity from a client-configuration incompatibility to P0. UploadPartCopy has no request body for the caller to checksum. The handler reads the source object, constructs an internal reader, and eventually enters the same PutObjectPart implementation. There is no client header and no SDK setting that can repair the request. Every checksum-enabled MPU therefore rejected UploadPartCopy by construction.
What AWS requires
This cannot be decided by saying that MinIO has historically behaved a certain way. The S3 protocol is the authority.
The AWS UploadPart API describes each algorithm-specific checksum header as something that “can be used as a data integrity check.” More importantly, its response fields say that the checksum is present only when it was provided in the request.
The AWS UploadPartCopy API is different: when the MPU was created with an algorithm, the copy result contains that part checksum. There is no copy request body, so this is necessarily a server-computed value.
The AWS ListParts API is the standard way to recover checksums for parts in an upload that is still in progress.
The algorithm/type matrix also rules out treating the repair as one Boolean flag:
| Algorithm | FULL_OBJECT |
COMPOSITE |
|---|---|---|
| CRC64NVME | Supported | Unsupported |
| CRC32 / CRC32C | Supported | Supported |
| SHA1 / SHA256 | Unsupported | Supported |
FULL_OBJECT is limited to CRCs that can be linearized, but SHA1 and SHA256 still need correct per-part digests for COMPOSITE completion.
SDK configuration makes the gap practical. Current AWS SDKs usually calculate request checksums when an operation supports them, but users can choose request_checksum_calculation = when_required, and low-level callers can initiate an algorithm without repeating it on every part. S3 accepts those requests; SILO did not.
Why removing the check is not a fix
The most tempting patch is to delete the comparison and allow a checksum-less part to proceed. That only moves the failure to completion.
SILO does not reconstruct and re-read all object bytes during MPU completion. It reads ObjectPartInfo.Checksums from each part.N.meta:
- a missing entry immediately becomes
InvalidPart; FULL_OBJECTcallsChecksum.AddPart, combining digests with their part lengths;COMPOSITEconcatenates the raw digest bytes and hashes them into the object checksum.
The actual invariant is therefore:
Deleting the upload check without filling the metadata would make UploadPart appear successful, leave ListParts incomplete, omit the UploadPartCopy response value, and fail later during completion. A delayed failure is harder to diagnose than the original immediate one.
Alternatives considered
| Option | Benefit | Fatal problem | Decision |
|---|---|---|---|
| Delete the strict check | Smallest diff | Part metadata still lacks the checksum; completion must fail | Rejected |
Relax only FULL_OBJECT |
Unblocks some default CRC clients | Leaves COMPOSITE and SHA incompatible; cannot close #46 |
Rejected |
| Re-read every part at completion | Avoids storing a digest during upload | Adds O(object size) second-pass I/O and still cannot fix ListParts or the copy response |
Rejected |
Always return the server value from normal UploadPart |
Makes federation forwarding easy | Violates the AWS response contract | Rejected |
| Copy the AIStor implementation exactly | Commercial precedent | CRC-only fallback and a transformed-stream placement risk | Rejected |
| Compute and persist in one pass over logical plaintext | Complete protocol behavior, no second I/O, CRC and SHA support | Requires an explicit plaintext checksum reader distinct from the storage reader | Accepted |
What the commercial edition taught us
We downloaded and verified the then-current MinIO AIStor RELEASE.2026-08-07T18-34-35Z. Without a commercial license the server enters offline mode and denies S3 operations, so the evidence came from Go pclntab and ARM64 disassembly, not a black-box compatibility run.
The static analysis showed that AIStor already:
- installs a server hasher when the client checksum is absent;
- persists the result in part metadata;
- exposes checksum fields in
CopyPartResult.
It nevertheless applies fallback only to CanMerge() algorithms—CRC32, CRC32C, and CRC64NVME. SHA1/SHA256 COMPOSITE still follows the old checksum missing path. More importantly, the hasher is attached in the object layer to the current r.Reader; under compression or encryption that reader may already represent transformed storage bytes.
AIStor validated the general direction—compute and store—but not an implementation that SILO could copy mechanically.
How adversarial review overturned the first design
The first plan tried to centralize every decision inside erasureObjects.PutObjectPart: read the MPU metadata in the object layer and install a server hasher when the incoming reader had no client checksum. It looked attractive because all internal callers would share one rule.
The first Fable 5 Max adversarial review found that this design was wrong for compression.
newS2CompressReader is not a lazy wrapper. Construction immediately launches a goroutine:
The S2 writer also reads several blocks concurrently. After constructing the compressor, the handler still performs option parsing, encryption preparation, and the object-layer call. By the time PutObjectPart installed a hasher, the plaintext reader could already have lost several MiB:
- a large part would get a checksum with a missing prefix;
- a small part could reach EOF before installation and produce no result;
- mutating
ServerSideHasherconcurrently withReadwould be a data race.
That finding changed the responsibility split:
The handler installs the hasher before any eager transform starts; the object layer validates the algorithm, requires a result, and persists it atomically.
This was the decisive turn in the design. Putting logic in the lowest layer may look more uniform, but stream correctness depends equally on when bytes begin moving and which representation of those bytes a layer can see.
Final implementation
A dedicated logical checksum reader
PutObjReader originally distinguished two concepts:
Reader, the stream sent to storage, possibly compressed or encrypted;rawReader, used by older ETag and checksum code.
Under compression, even rawReader may not directly see plaintext; it can merely carry an ETag through an etag.Tagger chain. The repair therefore did not overload it. It added an unexported field:
This reader always represents the logical S3 part bytes. WithEncryption can replace the storage Reader, but it must preserve checksumReader.
Unexported accessors on PutObjReader then:
- return the effective client or server checksum type;
- prefer the client value whenever it exists;
- otherwise return the server result finalized at EOF.
Keeping the mechanism unexported minimizes public Go API growth and gives #63 a shared internal path without prematurely changing ordinary CopyObject behavior.
Preparing the hasher before transformations
prepareMultipartChecksumReader loads the algorithm and checksum type saved with the MPU:
- no declared algorithm means no work;
- an existing client checksum is compared by base algorithm;
- a wrong algorithm preserves the
InvalidArgumentrejection; - an omitted client checksum installs the corresponding server hasher on the plaintext reader.
For normal UploadPart:
- the compressed path prepares
actualReaderafter request-checksum parsing but beforenewS2CompressReader; - the uncompressed path prepares the request hash reader before the encryption reader is constructed.
For UploadPartCopy:
- a checksum-enabled MPU first gets an inner hash reader over the logical source range;
- a range copy hashes only the selected bytes;
- compression and destination encryption start only after that reader is ready.
The object layer remains authoritative
Early handler preparation does not replace the storage invariant. erasureObjects.PutObjectPart still:
- re-parses the expected MPU algorithm;
- requires an effective checksum type that matches;
- obtains the checksum map after erasure encoding finishes;
- reports an internal error instead of committing if an enabled algorithm has no result;
- writes the checksum with the ETag, sizes, and index into
part.N.meta, then atomically renames the part.
An internal caller that bypasses the HTTP handler without preparing a valid checksum is therefore rejected just as before. It cannot silently commit a part that violates the MPU invariant.
CopyPart response shape
CopyObjectPartResponse gained the five algorithms supported by this source tree:
All are omitempty, so an MPU without checksums produces the old XML. Normal UploadPart still uses the existing TransferChecksumHeader and echoes only a client request value; fallback computation does not alter that response.
Why it works
After the repair, the data flow is:
This satisfies four requirements that previously appeared to conflict:
- Protocol compatibility: omitting an optional header succeeds.
- No integrity downgrade: a supplied client value is still checked end to end and is never hidden by server fallback.
- Correct object semantics: the checksum covers logical S3 bytes, not compressed data or ciphertext.
- Controlled cost: hashing shares the existing read and adds CPU, not a second disk or network pass.
EOF has a precise role. hash.Reader finalizes ServerSideChecksumResult only when it reaches EOF. Closing the compression pipe synchronizes the compressor goroutine with the storage read; the object layer reads the result only after encoding returns. Targeted -race tests verified that concurrency boundary.
The compatibility-baseline blocker
The five new CopyObjectPartResponse fields are exported Go API. SILO’s buildscripts/rebrand-guard rescans imports, environment variables, headers, routes, storage markers, and exported symbols, then compares them in both directions with buildscripts/rebrand-guard/compat-baseline.json. An unacknowledged symbol makes CI fail.
After recording the five #46 fields, the guard still reported two additions:
They did not come from #46. They belong to the earlier database-notification repair f1ba68358 on the local main branch. The cmd startup path intentionally needs the exported type for errors.As, but that earlier commit had not updated the compatibility baseline. Every later change based on that HEAD would therefore fail the CI guard.
We chose “option A”: acknowledge the two notification symbols as part of their original repair while retaining the five #46 fields. The final baseline diff is exactly seven additions and zero deletions, and the guard reports:
This does not disable the check. Exact set equality means that acknowledging a nonexistent symbol also fails. The change explicitly records two intentional compatibility-surface additions.
golangci-lint has not yet run locally; it remains a remote go.yml gate. Green local go test, go vet, race, and rebrand-guard results do not substitute for green remote CI.
Verification evidence
The new tests execute 76 subtests across:
- CRC32, CRC32C, and CRC64NVME
FULL_OBJECT; - CRC32, SHA1, and SHA256
COMPOSITE; - correct client checksums, wrong algorithms, and wrong values;
- absence of a server-computed checksum in normal
UploadPartresponses; - server values in
UploadPartCopyresponses andListParts; - a real 5 MiB + 1 KiB two-part full-object merge;
- zero-length parts and overwriting the same part number;
- a range copy whose SHA256 covers only the copied interval;
- single-drive and 16-drive erasure backends;
- default, versioned, compressed, encrypted, and compressed-plus-encrypted modes;
- explicit SSE-C and SSE-S3.
Local validation included:
All passed. Two subsequent Claude Code Fable 5 Max implementation reviews and the final acceptance review returned GO with no blocking findings.
Cost, risk, and release boundary
When a client omits its value, the server performs one additional hash over the part. CRC cost is small; SHA costs more CPU. Both share the read that already had to occur, without buffering an entire part in memory or adding a completion-time second pass.
During a rolling upgrade, old and new nodes may answer the same checksum-less request differently: a new node accepts it while an old node returns 400. ObjectPartInfo.Checksums did not change format, so stored data remains downgrade-readable, but client-visible behavior stabilizes only after all serving nodes have upgraded. The release note must call that out.
The original local implementation later merged and shipped in Server 20260903. Its historical local verification record does not establish the state of any production deployment.
Why two follow-ups remain separate
Adversarial review found two related but independent issues.
#63: CopyObject + compression
Ordinary CopyObject can also attach a server-side checksum to a transformed stream. It shares the root cause and the new checksumReader mechanism, but it is a different API with a different test matrix and rollback boundary. We chose a separate repair and require that PR to reuse this plaintext-reader contract instead of inventing a second abstraction.
#64: legacy federation
Legacy etcd federation turns UploadPartCopy into an ordinary remote UploadPart. Under the AWS response semantics preserved here, that remote request does not return a server fallback value, so the proxy may still lack the checksum required for CopyPartResult. A follow-up must independently choose between a remote-returned value and an ETag-verified ListParts fallback. It must not make all external UploadPart responses non-compliant merely to simplify an internal proxy.
Separating them does not abandon consistency. Consistency is maintained through one shared rule:
Every server-computed S3 checksum binds to the logical plaintext stream, is installed before any eager transform, and is validated and persisted by the object layer that owns the storage invariant.
Lessons retained
The repair leaves lessons more durable than its individual lines of code:
- An optional header does not make internal state optional. If the protocol lets the client omit a value, the server must produce the state its own completion path needs.
- Request acceptance and response disclosure are separate contracts. A normal UploadPart may compute internally and still omit the value; UploadPartCopy must return it.
- Stream layers are defined by byte semantics. The lowest layer is not automatically correct if it no longer sees logical bytes, and an eager goroutine turns “install later” into a race.
- A commercial implementation is evidence, not the specification. AIStor showed the direction and the boundary that could not be copied.
- A compatibility guard is a change-acknowledgment mechanism.
compat-baseline.jsonexists to assign every new compatibility surface, not merely to make CI quiet. - Independent defects should ship independently while sharing invariants. #63 and #64 remain separate, but both must cite and obey the checksum-reader contract established here.
The final result is not a broad relaxation. It is a stricter and more accurate boundary: clients may omit optional information; the server may not omit correctness.
4.26 - BadDigest, InvalidRequest, and the CompleteMultipartUpload Checksum Contract
The repairs for #48 and #50 are included in Server 20260903. This record supersedes the August proposal to leave CRC64NVME canonicalization unchanged.
Why validation belongs at completion
Initiation selects the checksum algorithm and object type; each part records
its checksum. Completion must validate the final value and any explicit type
assertion against that stored contract. A request cannot change COMPOSITE to
FULL_OBJECT just because both share the same base algorithm, or bypass the
assertion by omitting the final digest.
The repair distinguishes an omitted type from an explicit type and normalizes only internal representation flags. It uses operation-specific error types, leaving UploadPart and the global checksum-mismatch mapper unchanged.
Current error contract
All failures below return HTTP 400 and do not commit a new completed object.
| Request condition | S3 error |
|---|---|
| Wrong full-object or composite object digest | BadDigest |
| Explicit supported type differs from the initiated type, including a type-only assertion | BadDigest |
| Unknown or lowercase type token | InvalidArgument |
| Completion algorithm differs from the initiated algorithm | InvalidArgument |
| Missing required composite part checksum | InvalidRequest, naming the algorithm and part |
| CRC64NVME with COMPOSITE at initiation | InvalidArgument |
| CRC64NVME value with COMPOSITE rejected by checksum parsing at completion | InvalidArgument |
| Bare COMPOSITE type at completion of a CRC64NVME FULL_OBJECT upload | BadDigest |
| Incorrect client checksum during UploadPart | Existing XAmzContentChecksumMismatch |
For FULL_OBJECT, part checksums may be omitted; any supplied values remain
validated. Matching type-only assertions are allowed. An omitted optional type
does not assert COMPOSITE. SHA1/SHA256 with FULL_OBJECT are rejected by the
algorithm/type parser before the stored-type comparison.
CRC64NVME decision and source evidence
The original review deferred #50 pending evidence; that is historical, not the current contract. PR #93 rejects the invalid algorithm/type combination in headers and trailers, and PR #96 removes completion’s canonicalization. Both precede the 20260903 tag. Completion can fail at either parsing or stored-type comparison, which explains the two distinct error codes above.
The initial mapping is in PR #74, with
the explicit type follow-up 7e079ff05 merged through
PR #85. Current
handler tests
exercise full-object and composite mismatches, type-only assertions, invalid
tokens and both CRC64NVME rejection stages. This is committed regression
coverage, not a claim that this documentation update reran all Server tests.
Separate streaming-checksum follow-up
PR #143, following @cbornet’s
#107, handles an aws-chunked request
that advertises x-amz-trailer but supplies its checksum in a header, as used by
the AWS Java SDK v2. It also rejects an invalid header-delivered value instead
of dropping validation. This later repair is on main and not in Server
20260903. It is separate from completion error mapping.
Compatibility and remaining boundaries
Callers that inspect error codes now see checksum/type failures as BadDigest
and missing composite values as InvalidRequest. Successful checksum-free
uploads and ETag semantics are unchanged. No metadata migration is required.
Several narrower differences remain: providing a final digest when initiation
recorded no checksum is rejected as BadDigest; composite part-count and value
mismatches share one description; a -N suffix on a full-object checksum is
not itself validated as a part count. These are documented observations, not
claims of separately filed public issues or complete AWS parity.
4.27 - ListMultipartUploads: Implementation, Upgrade Contract and Design History
This is the problem, design, and decision record for SILO issue #79.
September 16 implementation and upgrade contract
PR #198 retains mr javad seydi’s original metadata-and-scan contribution and adds maintainer fixes for disappearing markers, incomplete discovery, cancellation confirmation and upgrade diagnostics. The pre-release compatibility follow-up restores legacy as the default and makes strict mode an explicit process-environment opt-in. This section describes those source changes. It is not included in Server 20260903, and does not establish production performance or deployment acceptance. The August 30 analysis below remains a historical design record.
Listing and cancellation
New uploads persist their bucket and object key in the existing xl.meta; completion removes these upload-only fields. Strict listing discovers durable uploads across pools and sets, verifies metadata with the existing read quorum, and applies prefix, delimiter, CommonPrefixes and a global page limit of at most 1,000. Restarting a node or choosing another endpoint does not depend on rebuilding its upload cache.
Ordering is (key, initiation time from the native upload ID, encoded upload ID). A returned marker defines this boundary even after the corresponding upload is completed or canceled. This supports clients that echo the server’s markers; it does not promise arbitrary lexical comparison of random upload IDs. There is no snapshot across pages under concurrent mutations. Without key-marker, upload-id-marker is ignored. With a key marker, invalid base64 retains the existing 404 response; a decodable but unsupported native ID returns 400. Unsupported persistent IDs are legacy records, never assigned a fabricated current initiation time.
Directory discovery requires floor(N/2)+1 successful drive scans per set. For example, two available drives out of four are insufficient and return 503, even when metadata could still be read from two copies. A source-drive identity read may exclude another bucket only after validating its bucket/key hash; uncertain identities require a quorum read. Strict mode returns MultipartListingNotReady (503) for legacy records and MultipartListingMetadataInvalid (503) for invalid identities. It never silently switches the whole request to cache-based listing.
In default legacy mode, Abort retains the released read quorum, best-effort cleanup and pool-order return behavior, avoiding new 503 responses for previously successful cancellations. In strict mode it checks every relevant pool and requires floor(N/2)+1 deletion acknowledgements per set. Remnants below read quorum can still be retried. When a majority was already absent but remnants were observed, those observed copies must be cleaned successfully; successful responses from empty drives cannot mask their deletion failures. Unknown pools or insufficient confirmations still return 503.
In both modes, an Abort with the wrong key or bucket cannot evict another valid upload’s cache entry. The S3 HTTP layer retains its existing idempotent response: a nonexistent upload also returns 204; malformed input, authorization and quorum errors remain errors. 204 does not prove that every physical copy has been deleted. Offline part data may still need later cleanup.
Unresolved creation-write boundary: these confirmations do not fence a physical creation write that continues after its caller receives a storage timeout. A fault-injection test reproduces a 16-drive/EC:8 case where seven delayed writes and seven offline old copies restore a writable upload after acknowledged cancellation. The test preserves this known limitation; its passing status is not a repair claim. A durable creation fence needs a separate storage-consistency design.
Coordinated upgrade
The default is legacy, retaining the old exact-key/cache listing limitations. An ordinary upgrade does not require pausing production or draining uploads to enable the new listing. The mode is read only from the server process environment, MINIO_API_MULTIPART_LISTING=legacy|strict; no shared configuration key is added. An unset value selects legacy. An invalid value logs a diagnostic and falls back to legacy while preserving other API settings. Do not set this mode with mcli admin config set.
Only before opting into strict mode must operators upgrade all writers, stop introducing old-format uploads, finish or abort legacy uploads using known keys/IDs, run the read-only preflight below, and verify scan capacity against their workload. Once these checks pass, set MINIO_API_MULTIPART_LISTING=strict in every server’s service environment and restart; check the effective mode at each endpoint. Restart paginated traversals after changing modes. Issue #79 remains open for the default listing limitations; this batch does not claim a complete repair.
If a development build already persisted api multipart_listing, back up configuration and record API values, upgrade all configuration writers to the patched version, prevent concurrent configuration writes, then remove only that historical key:
The patched server ignores the historical key’s mode value without automatically rewriting shared configuration or deleting history. The config get/export views omit retired keys, so their absence from those views alone does not prove deletion. Require a successful targeted reset, then verify preserved values after restart and during a controlled rollback check; do not reset the whole api subsystem. A development build can reintroduce the key on its next configuration write, and history containing that key should not be replayed directly. Existing API values can temporarily be pinned through their corresponding environment variables where needed, but that does not replace persistent cleanup. This procedure covers this API configuration change only; other rollback constraints require separate checks.
The read-only, SigV4-authenticated endpoint GET /minio/admin/v3/multipart-preflight requires admin:StorageInfo. For example, with credentials and an endpoint supplied by the operator:
The report contains mode, ready, complete, scannedEntries, legacyUploads, and per-pool/set drive coverage, uncovered drive indexes and oldest legacy initiation time. It bypasses upload caches, inspects even suspended pools and detects minority legacy copies. ready=true requires every drive to be inspected, no unreadable candidate metadata and no observed legacy copies. It cannot attest that every writer has been upgraded or that no concurrent writer will introduce a legacy record. Offline drives, scan errors, timeouts and budget exhaustion prevent readiness; an incomplete count is not a zero count. Rerun after drives return and before enabling strict mode.
For uploads whose original key/ID has been lost, the existing stale-upload cleanup scans each server’s local drives. Age is measured from creation, not recent part activity. Retain the existing cleanup policy, wait and verify actual drain; the default 24-hour expiry and 6-hour interval do not guarantee drain completion. Do not shorten expiry to accelerate an ordinary upgrade, since this can also remove active long-running uploads. Continue using default legacy mode if drain cannot be established. This batch changes neither cleanup policy nor the available deletion APIs.
The shared maxUploadsList cap changes from 10,000 to 1,000 in both modes; clients that assumed one response contained everything must paginate.
A legacy upload lacks bucket/key identity and cannot safely be excluded as belonging to another bucket, so it can block strict listing for any bucket. A valid new-format identity belonging to another bucket can be filtered earlier. This does not impose deployment-wide 503 responses on default legacy listing.
Scan capacity and evidence limits
Each process admits two scans, with 16 identity workers and four full-metadata workers per scan. Directory reads pass a finite count with overflow detection. The aggregate budget is 100,000 returned directory entries, including repeated entries on different drives and hash directories; it is not a promise to list 100,000 unique uploads. Concurrent directory calls may already be in flight when the aggregate budget is exceeded. Overflow returns SlowDown (503), never a successful partial page. A 30-second context budget stops further scheduling; admission stays held until scan workers exit. This is neither a precise memory ceiling nor a guarantee that a canceled physical system call stops immediately.
As a budget illustration only, one upload per unique key on every drive of an N-drive set costs about 2 × N entries across the two directory levels: 100000 / (2 × N) uploads, or about 3,125 at N=16. Multiple uploads under one key share the hash-directory cost; other sets/pools, stale directories and timeouts also matter. This is not a universal upload-count limit.
Every page still rescans durable state: total enumeration cost grows with both stored candidates and page count. The tests cover missing markers, multi-pool coverage, partial-deletion retries, identity fallback, RPC directory bounds, cancellation admission and the known late-write counterexample. Temporary multi-node and maintained-client checks establish functional behavior for their recorded environment. Production-scale latency and foreground-load impact remain deployment-specific acceptance work; merging the source does not certify them.
A September 16 temporary Docker Desktop arm64 run used two nodes, four APFS-backed bind volumes and 11,000 uploads across two buckets, while other local validation was running. One 1,000-entry page for the 10,000-upload target bucket took 18.7 seconds; two concurrent requests returned retryable SlowDownRead responses after about 25–27 seconds. These observations do not meet the provisional five-second page target and are not an isolated SSD benchmark. Capacity and foreground-load acceptance remain open; the bounded scanner must not be advertised as a large-scale performance fix.
Historical analysis (2026-08-30): “current”, “recommended” and gate language below describes that review and proposal. The September 16 section above defines the implementation and remaining limits. Capability advertisement and the old one-day drain estimate are not current guarantees.
The problem in plain language
Imagine that four large files are still being uploaded:
An S3 client asks, “show me every unfinished upload below tables/.” AWS S3 returns the first three. SILO currently treats tables/ as if it were the complete name of one object, looks for exactly that object, and returns an empty list.
If the client removes the prefix and asks for every unfinished upload in the bucket, SILO takes a different shortcut: it reads a process-local memory cache. That cache may contain all four uploads on the node that created them, but it does not survive a restart and is not authoritative across nodes. The upload data is still on disk; the list is wrong.
This is why the defect is more serious than one ignored query parameter. Cleanup tools can receive 200 OK, conclude that no unfinished uploads exist, and report success while uploads remain on disk. The server is not losing committed objects, but it is giving callers a false view of unfinished work.
Executive decision
SILO should fix this behavior if it intends to keep advertising practical S3 compatibility.
The repair is justified because the current endpoint silently claims success, behaves differently after restart or node switching, and breaks standard prefix-based cleanup and pagination. The default 24-hour stale-upload collector limits storage accumulation on default configurations, but it does not make the API result truthful.
The repair is not a small change. Existing upload directories contain only a one-way hash of the bucket and object key, and the original key is not stored in their xl.meta. A correct implementation must begin persisting that identity for new uploads, discover candidates with erasure-aware quorum rules, apply S3 semantics globally across pools and sets, and handle legacy uploads during a rolling upgrade.
The recommended direction is therefore:
- record the bucket and object key in the upload’s existing quorum-written metadata;
- build a bounded on-demand scan as the durable correctness path;
- keep any cache only as a rebuildable optimization;
- enable strict S3 behavior only after every writer has upgraded and all keyless legacy uploads have drained;
- consider a durable secondary index only if measurements prove that scanning cannot meet a product-approved service-level objective.
Sources and provenance
The problem statement and proposed design are grounded in five kinds of evidence.
The S3 contract
The AWS ListMultipartUploads API defines the public contract for general-purpose buckets:
prefixselects every upload whose key starts with that string;delimitergroups matching keys intoCommonPrefixes;max-uploadslimits a page, with 1,000 as the documented maximum;key-markerandupload-id-markercontinue a truncated listing;upload-id-markeris ignored whenkey-markeris absent;- results are ordered by object key, then by initiation time for uploads with the same key.
AWS documentation does not settle every implementation edge unambiguously. Equal timestamps, invalid or out-of-range max-uploads, URL encoding, marker boundaries, and the way CommonPrefixes consume a page should be captured once against AWS and stored as fixtures before implementation.
The reported defect
Issue #79 supplied a self-contained signed reproducer against pgsty/silo:latest and compared SILO with AWS, RustFS, SeaweedFS, and Garage. Its four central observations reproduce:
| Request | Required behavior | Observed SILO behavior |
|---|---|---|
prefix=t/ |
return the three keys beginning with t/ |
returns no uploads |
max-uploads=1 |
return one item and continuation markers | returns every cached upload |
key-marker=t/a_b/p2 |
continue after that key | returns every cached upload |
prefix=t/&delimiter=/ |
return grouped CommonPrefixes |
returns neither uploads nor prefixes |
The issue correctly identifies a compatibility failure, but its statement that max-uploads is always ignored and IsTruncated is always false is broader than the implementation. Those claims hold on the empty-prefix cache path used by the reproducer; the exact-object path can honor max-uploads and upload-id-marker and can set IsTruncated.
The upstream design history
The behavior was inherited rather than invented by SILO:
- MinIO PR #5248 deliberately removed prefix-based listing from the erasure backend in 2017 “to simplify” multipart support.
- MinIO PR #20407 added the empty-prefix multipart cache in 2024, mainly for Alluxio tests.
- A 2025 report of the same exact-key behavior, MinIO issue #20989, was closed as working as intended.
- SILO’s current S3 compatibility reference already records the exact-object-name divergence, although it did not explain the cache, pagination, marker, delimiter, or restart limitations before this design record.
This history explains why the code looks deliberate. It does not make the endpoint compatible with the AWS contract.
Source review
The current source has two mutually exclusive listing paths:
The important locations are:
cmd/erasure-server-pool.go: empty-prefixmpCache, per-pool concatenation, and the internal exact-object lookup used byNewMultipartUpload;cmd/erasure-multipart.go: exact-object listing, upload directory construction, stale-upload cleanup, and the quorum write for a new upload’sxl.meta;cmd/erasure-sets.go: hashing a supplied object name to one erasure set;cmd/bucket-handlers.go: public request validation, including a501 NotImplementedguard whenkey-markerdoes not share the request prefix;cmd/object-api-multipart_test.go: a large expected-results table whose final assertion block checks only echoed scalar fields, not the returned uploads, prefixes, markers, or truncation state.
Independent reproduction and adversarial review
The issue scenario was independently reproduced against the reviewed SILO source with a single-node server and SigV4 requests. Additional probes established that:
- an exact object key can paginate its own uploads;
upload-id-markercurrently affects that exact-key path even withoutkey-marker, contrary to AWS;NextKeyMarkerremains empty on an exact-key truncated page;max-uploads=0behaves as unlimited in the current path;- a plain server restart empties the bucket-wide view while exact-key lookup still finds the on-disk uploads.
A second adversarial architecture review challenged the storage, quorum, migration, suspended-pool, mixed-version, and performance assumptions. The corrections from that review are incorporated below; this record does not treat an AI review as a substitute for code tests or an AWS conformance capture.
What the code actually does
Empty prefix: a volatile node-local view
With no prefix, the pool layer returns every MultipartInfo for the bucket from mpCache, sorted only by initiation time. It does not apply max-uploads, key-marker, upload-id-marker, or delimiter, and it does not compute continuation markers or IsTruncated.
The cache is initialized empty at process startup. Creation populates only the node that handles the request. Completion and abort delete cache entries, including peer notifications in some paths, but creation has no equivalent durable cluster-wide population or startup rebuild. Consequently:
- a restart can change a non-empty listing into an empty one;
- two nodes can return different answers for the same bucket;
- a successful response is not evidence that the server has enumerated durable upload state.
Non-empty prefix: an exact object lookup
With a non-empty prefix, the string is passed through object hashing as though it were a complete object name. One erasure set is selected, and listing reads the directory derived from sha256(bucket/object).
This path can enumerate multiple upload IDs for that exact object. It sorts them by initiation time, applies its upload-id-marker, stops at max-uploads, and sets IsTruncated. It still does not implement lexical prefix matching, CommonPrefixes, general key-marker semantics, or NextKeyMarker.
Multiple pools make pagination less correct
For a non-empty request on a multi-pool deployment, the pool layer invokes each active pool with the same maximum and concatenates the results. It does not perform a global ordered merge or recompute page boundaries and next markers. A request for N items can therefore collect up to N from each pool.
Suspended pools are skipped by listing and by the other public multipart verbs. In-progress uploads left on a suspended or decommissioning pool are therefore inaccessible, not merely unlisted. That is a related lifecycle defect, but listing alone must not advertise handles that PutObjectPart, ListParts, CompleteMultipartUpload, and AbortMultipartUpload cannot use. Pool drain or forced abort should be designed as a separate cross-verb change.
Why current uploads cannot be backfilled
The multipart namespace is flat:
The hash is one-way. The original bucket and key are not encoded in the path. They are also not stored as a name field in the current multipart xl.meta; the supplied object name only influences the erasure distribution during newFileInfo construction.
Therefore an all-directory scan can discover that an upload exists, but it cannot determine which bucket or key it belongs to. The current node-local cache cannot repair this reliably because it is incomplete across nodes and disappears on restart.
This rules out a tempting “small” fix: scanning every existing xl.meta and applying prefix filters. New identity metadata or a durable index is required, and old keyless uploads need an explicit migration policy.
Complexity assessment
The semantic algorithm is not the hardest part. The hard part is obtaining a complete, quorum-valid, globally ordered input set without turning a listing call into an uncontrolled cluster-wide metadata storm.
| Area | Complexity | Why |
|---|---|---|
| Pure S3 filtering and pagination | Medium | Rules are finite, but marker and delimiter edge cases need captured AWS evidence. |
| Persisting bucket/key in new upload metadata | Medium | It reuses an existing quorum write, but completion, rollback, healing, and replication compatibility must be tested. |
| Candidate discovery | High | The namespace mixes every bucket and duplicates each upload across erasure drives. One-disk discovery can miss quorum-valid uploads. |
| Quorum and concurrent deletion | High | A scan must reject minority ghosts while tolerating abort, completion, GC rename-to-trash, and transient ENOENT. |
| Multi-pool global pagination | High | Results must be merged, sorted, truncated, and marked once across all accessible pools and sets. |
| Rolling migration | High | Old writers keep creating keyless uploads; old completers may preserve unknown internal metadata. |
| Performance and resource control | High | A bucket request may require inspecting every active upload in the cluster, not only that bucket. |
Overall, this is a high-complexity compatibility project with medium wire-compatibility risk and high implementation-correctness risk. It is not a destructive object-format migration: the recommended design adds internal metadata for new incomplete uploads and leaves the existing directory scheme in place.
Compatibility and operational impact
Wire behavior changes
A correct implementation deliberately changes observable results:
prefix=foowill matchfoo,foobar, andfoo/..., not only the exact keyfoo;- bucket-wide results will be ordered by key and initiation time rather than only initiation time;
max-uploadswill actually limit a page;- the default and maximum will move from SILO’s current 10,000 constant toward the AWS limit of 1,000, subject to the captured edge-case contract;
- clients must follow
NextKeyMarkerandNextUploadIdMarkerinstead of assuming one response contains everything; - delimiter requests will return
CommonPrefixes; - the current handler-side
501for a marker outside the prefix will be replaced by the captured AWS semantics.
These are compatibility fixes, but they can break software that accidentally depends on SILO’s old non-S3 behavior. In particular, a client that ignores pagination may see fewer entries after the repair. Strict behavior should therefore be introduced through an explicit release and rollout contract, not silently slipped into an unrelated patch.
Storage-format compatibility
The recommended write path adds the bucket and object key as reserved internal metadata inside the new upload’s existing quorum-written xl.meta. It does not rename multipart directories or create a second transactional write.
Before CompleteMultipartUpload renames upload metadata into the completed object, the new upload-only fields must be removed alongside the multipart checksum fields that are already stripped there.
An old binary completing an upload created by a new binary will not know to remove the new internal keys. They would remain inert and hidden from S3 user metadata, but persist in the completed object’s internal metadata. Rolling-upgrade tests must prove that unknown reserved keys do not disturb healing, replication, metadata comparison, or downgrade reads. The product must then choose between tolerating that residue and adding a scrubber; it must not assume the keys disappear.
Operational cost
Because all buckets share one flat hash namespace, an on-demand scan is O(all active multipart uploads in the cluster), not O(uploads in the requested bucket). Bounded parallelism, cancellation, memory limits, and failure behavior are part of correctness, not optional tuning.
Default SILO configuration expires stale multipart uploads after 24 hours and runs cleanup every 6 hours. Once the last old writer has been upgraded, the keyless population should normally drain within roughly 30 hours. Operators with a larger custom expiry have a longer migration window. A zero value is mapped back to the 24-hour default in the current code; no supported “disabled” expiry value was identified in this review.
The collector bounds default storage accumulation, but does not repair a false listing response. It also does not remove the need to test sustained legitimate multipart activity, failure modes, and custom expiry settings.
Severity
The recommended classification is P1 / high compatibility, not P0:
- no committed object data loss was demonstrated;
- no security boundary is bypassed;
- unfinished uploads remain on disk until completed, aborted, or collected;
- default stale-upload cleanup bounds accumulation in the ordinary configuration.
It remains high rather than medium because the server returns fabricated success, the answer changes after restart or node switching, and cleanup or quiescence tooling can be misled into false confidence.
Options considered
Option 0: leave the current behavior unchanged
This has no engineering cost and preserves every accidental behavior. It also preserves false 200 OK responses, node-local inconsistency, restart volatility, broken prefix cleanup, and an inaccurate impression of S3 support.
This option is acceptable only if SILO deliberately downgrades the public compatibility claim and treats the endpoint as unsupported. Even then, silently returning an incomplete success is inferior to explicit rejection.
Decision: reject as a long-term position.
Option 1: explicit documented divergence
Reject combinations that SILO cannot honor with a stable NotImplemented-class error and document the exact supported subset. This is operationally honest and much smaller than full compatibility.
It is still a breaking change: tools that currently receive an empty or unbounded 200 OK may begin failing jobs. It also does not produce an S3-compatible endpoint. The error behavior and default release policy must be deliberate.
Decision: acceptable short-term containment if full compatibility is declined or deferred; not a compatibility fix.
Option 2: persist identity, scan durable state, optionally cache
For each new upload, store the bucket and key in reserved internal metadata in the upload’s existing xl.meta. For listing, discover upload directories across accessible pools and sets, validate candidates with erasure read quorum, then run one global S3 semantic layer. A cache may accelerate this path only if it can be rebuilt and reconciled from durable state.
This avoids a second write transaction and keeps the directory layout stable. Its principal cost is the cluster-wide scan.
Decision: recommended, subject to a performance and failure-mode spike.
Option 3: durable bucket-scoped ordered index
Maintain a secondary index ordered by bucket, key, and upload identity. Listing becomes scalable and naturally paginable, but create, complete, abort, healing, rollback, and reconciliation must keep two locations consistent across failures. The design resembles multipart index structures that upstream MinIO deliberately removed while simplifying this subsystem.
Decision: no-go unless measurements show that Option 2 cannot meet the product-approved service-level objective.
Rejected variant: repair only mpCache
Filtering, sorting, paginating, broadcasting creates, or rebuilding the current cache would improve symptoms but would not by itself establish a durable quorum-valid source of truth. A cache-only patch risks producing a more convincing but still incorrect answer.
Decision: reject. A cache can optimize a correct read path, never define it.
Recommended design
1. Freeze the public contract first
Create a recorded AWS fixture suite for general-purpose buckets covering:
- ordering across keys and multiple uploads of one key;
- equal initiation times and a deterministic total-order tie-break;
- prefix and exact-key overlap;
- key-marker with and without upload-id-marker;
- upload-id-marker without key-marker;
- delimiter,
CommonPrefixes, and page accounting; max-uploadsomitted, 0, 1, 1,000, and greater than 1,000;encoding-type=url;- empty pages, final pages, and next-marker values.
The captured responses should become repository fixtures. CI should not depend on live AWS access.
2. Persist recoverable identity in the existing write
At NewMultipartUpload, add reserved internal metadata for the canonical bucket and object key before the existing writeAllMetadata quorum write. The exact key names are an implementation detail, but they must be versioned, unambiguous, size-bounded by the existing object-key limits, and excluded from client-visible metadata.
At successful completion, delete those upload-only keys before copying fi.Metadata into the final object metadata and before renameData. Abort and stale cleanup already delete the entire upload directory and need no separate index operation.
3. Separate discovery from validation
Candidate discovery and candidate validity are different questions.
For every accessible, non-suspended pool and set:
- list candidate hash and upload directories from all online drives required by the configured list-quorum policy;
- union and deduplicate those names;
- read the candidate
xl.metathrough the normal erasure metadata machinery; - include the upload only when its metadata is quorum-valid and contains a valid bucket/key identity;
- tolerate a candidate disappearing during abort, completion, or stale cleanup;
- under strict list quorum, fail the request rather than return a partial
200 OKwhen a required set cannot be evaluated.
Using the first healthy disk for discovery is insufficient: that disk may have been offline when a still-quorum-valid upload was created.
4. Apply semantics once, globally
Feed the validated candidates from all pools and sets into a pure semantic layer. The layer owns bucket filtering, prefix, delimiter grouping, ordering, markers, maximum-page accounting, URL encoding, IsTruncated, and next markers.
Pool-local limits and markers must not be applied before the global merge. The result should be deterministic under duplicate discovery and independent of which node handles the request.
5. Preserve the internal exact-object operation
erasureServerPools.NewMultipartUpload currently calls ListMultipartUploads(bucket, object, ...) to keep another upload for the same object in the same pool. If the public function starts treating that argument as a lexical prefix, foo could match foobar and select the wrong pool.
Introduce a narrowly named internal helper such as FindMultipartUploadPool or ListMultipartUploadsExact. It should use the existing object hash path and must not share the public prefix semantics.
6. Treat cache as an optimization
The existing mpCache may be removed. If retained, it must satisfy all of the following:
- durable state remains authoritative;
- startup can rebuild it;
- create, complete, and abort updates are propagated consistently;
- reconciliation detects missed events and stale entries;
- a cold or divergent cache falls back to the quorum-valid scan;
- correctness tests pass with the cache disabled.
7. Gate strict behavior through rolling migration
Legacy upload records lack bucket/key identity and cannot be reconstructed reliably. Use two externally meaningful modes:
- legacy mode, the initial upgrade default: new writers persist identity; keyless uploads are counted and drained; the documented response policy for a mixed keyed/keyless population must be selected explicitly;
- strict mode: activation requires every writer node to advertise the new metadata capability and the observed keyless count to be zero. Discovering a keyless upload afterward is an error with anomaly telemetry, not a silent omission.
A short shadow comparison can help validate the new scanner, but a permanent third operating mode is unnecessary unless the spike finds a need. With default expiry, the expected legacy drain is about one day plus one cleanup interval after the last old writer stops.
There is one unresolved product choice in legacy mode:
| Policy | Advantage | Cost |
|---|---|---|
| return the complete keyed subset with documented telemetry | keeps tools operating during the bounded drain | still returns an incomplete 200 OK that ordinary clients cannot see is incomplete |
| fail listing while any keyless upload exists | never fabricates completeness | can block cleanup and existing jobs throughout the drain window |
This choice belongs in the ADR. Strict mode has no such ambiguity: it must fail loud if its precondition is violated.
8. Keep suspended-pool lifecycle separate
Listing should initially mirror the accessibility contract of the other multipart verbs and scan non-suspended pools. Adding suspended-pool entries to listing alone would expose uploads that cannot be extended, completed, or aborted.
Open a separate lifecycle design for in-progress uploads when a pool drains: either keep all multipart verbs available until the uploads finish, migrate them, or force-abort them under a documented policy. Do not hide that problem inside #79.
Performance spike and decision rule
Option 2 is preferred because it has one durable write location, but its scan cost must be measured rather than assumed.
Generate 1,000, 10,000, and 100,000 active uploads across a matrix of pools, sets, and drive counts. Measure:
- cold and warm p50/p95/p99 latency;
- total and per-drive
ListDiroperations; - metadata-read and internode RPC counts;
- peak memory and allocation volume;
- cancellation latency;
- behavior with slow, offline, healing, and intermittently disappearing drives;
- simultaneous create, complete, abort, and stale cleanup;
- first-page and deep-page cost with selective and empty prefixes.
The acceptance threshold is a product decision and must be recorded before interpreting the result. A guessed one- or two-second target is not evidence. If the scan meets the approved target with bounded resource use, reject Option 3. If it does not, use the measurements to design the smallest durable index that solves the demonstrated bottleneck.
Test and release gates
Semantic and unit tests
- pure table tests generated from recorded AWS fixtures;
- ordering, marker, delimiter, encoding, truncation, and maximum-edge coverage;
- property tests ensuring pagination returns each logical upload exactly once;
- deterministic behavior with duplicate candidates and equal timestamps.
Object and handler tests
- strengthen the existing object-layer table to assert uploads, common prefixes, markers, and truncation;
- parse and validate handler XML bodies instead of checking only status codes;
- verify default and invalid
max-uploadshandling; - test exact-helper pool selection independently of public prefix semantics.
Distributed and failure tests
- restart equivalence and node-switch equivalence;
- multiple sets and pools with a single global page boundary;
- candidate missing from one drive but present at quorum;
- minority ghost after partial abort;
- concurrent completion and GC rename-to-trash;
- unavailable set under every supported
list_quorumpolicy; - rolling upgrade, old-writer reintroduction, downgrade completion, and strict-mode gating;
- unknown internal metadata under healing and replication.
Delivery gates
- approve the ADR, including product mode and performance SLO;
- commit the captured conformance fixtures;
- complete and review the storage spike;
- implement and pass focused, full, race, and failure QA;
- update the S3 compatibility reference and operational guidance;
- commit and merge the source change;
- build and identify the release artifact or container image;
- canary a rolling upgrade and observe keyless-drain telemetry;
- enable strict mode only after its gates hold;
- verify the live endpoint before closing #79.
Passing an earlier gate is not evidence that a later gate happened.
Final recommendation: fix it, but do not rush it
Leaving the current endpoint indefinitely is the wrong trade-off. This is not an obscure response-field mismatch: it affects discovery and cleanup of unfinished data, returns successful but false answers, and changes behavior across nodes and restarts. Those properties undermine the practical meaning of S3 compatibility.
At the same time, a direct implementation patch is also the wrong trade-off. The current disk layout cannot identify legacy uploads, a correct scan needs erasure-aware discovery and quorum, and wire-correct pagination changes observable client behavior.
The balanced decision is:
- GO for the ADR, AWS fixture capture, metadata-plus-scan prototype, and performance/failure spike;
- GO conditionally for Option 2 after the product SLO and legacy response policy are approved;
- NO-GO for a cache-only repair, an immediate durable secondary index, strict-by-default behavior in a patch release, or closing the issue before rolling-upgrade reachability is demonstrated;
- if implementation capacity is unavailable, GO for an explicit documented divergence and stable error behavior rather than continuing to fabricate successful listings.
This preserves compatibility discipline without pretending that a high-risk distributed listing change is a two-line bug fix.




