Skip to content

This is the multi-page printable view of this section. .

Return to the regular view of this page.

PGSTY SILO Blog

News, Release & Security Notes

1 - Posts

Essays and analysis about MinIO, S3-compatible storage, and the SILO community fork.

Essays and analysis about MinIO, S3-compatible object storage, and the SILO community fork.

1.1 - MinIO Fork, Promise Kept

Two months ago in “MinIO is Dead, Long Live MinIO,” I promised I’d keep the MinIO fork patched. The recurring objection on HN is fair: can one person actually maintain something like this? The real answer isn’t clicking fork. It’s what happens when CVEs start landing.

Between April 15 and 17, pgsty/minio shipped RELEASE.2026-04-17, closing four CVEs and a handful of related vulnerabilities disclosed in the same window.

gh-release.webp

The scope I committed to originally was narrow: no new features, keep the supply chain running, handle reproducible bugs and security issues as they come in. This release is what it looks like when that promise gets tested.


What happened upstream

In December 2025, MinIO moved the open-source repo to maintenance mode. The README said security fixes would be “evaluated case by case.” In February 2026, the repository was archived and the landing page became “this repository is no longer maintained.”

The SECURITY.md in that same archived repo still says: “we will always provide security updates for the latest release.”

Over the past month, four high-severity and two medium-severity vulnerabilities have been disclosed against the final open-source release.

policy.webp

It’s been 184 days since the last upstream release. Vulnerabilities get disclosed; fixes ship only in the commercial build. The guidance for OSS users is a single line: upgrade to AIStor.

AIStor starts around $100k/year for 400 TiB — roughly S3 pricing, for software you install and operate yourself.

It’s a clean arrangement: archive the repo so there’s no obligation to patch, keep publishing CVE advisories for visibility, and route everyone who reads them toward the commercial product.

Someone still has to patch the old one.


What this release fixes

Full write-ups, CVSS arithmetic, and PoCs are in the release notes. The short version:

  • CVE-2026-33322 (OIDC JWT algorithm confusion, CVSS 9.8): under certain IdP configurations, an attacker who knows the OIDC ClientSecret can mint a token claiming any identity — including consoleAdmin — and MinIO will accept it. Vulnerable window: November 2022 through March 2026. About three and a half years.
  • CVE-2026-33419 (LDAP STS enumeration and brute-force): the login endpoint leaks which usernames are real, and there’s no rate limiting on the subsequent password guessing. End of the chain is an STS credential.
  • CVE-2026-34204 (replication-header metadata injection): a regular PUT or COPY with certain X-Minio-Replication-* headers can write an object into a permanently unreadable state. The data is still on disk; you just can’t read it back out.
  • CVE-2026-39414 (S3 Select memory exhaustion): one request, one OOM.
  • GHSA-hv4r-mvr4-25vw / GHSA-9c4q-hq6p-c237: two signature-verification bypasses on the unsigned-trailer path. Anonymous or forged-signature requests can successfully write objects on certain routes.

Plus the usual dependency cleanup from go-jose, go.opentelemetry.io, and the Go 1.26.2 upgrade itself — about twenty security items in total counting transitive dependencies.

issue.webp

How it got fixed

I said in the earlier post that I’d rely on AI coding agents, and that’s how this round went. My role was closer to “review and decide” than “write code.”

Per-issue flow, roughly:

  1. Codex drafts first. Given the CVE description and relevant code paths, it produces an initial patch.
  2. Claude Code reviews adversarially. Picks holes from the attacker’s side.
  3. Back to Codex. If it agrees with Claude Code’s critique, it reworks. If not, it has to write out why. No silent overrides.
  4. Another round of review by Claude Code, with both sides’ reasoning on the table. Iterate until they converge.
  5. Tests. Codex proposes cases, Claude Code adds more, Codex runs them, Claude Code reviews the results.
  6. I decide. Read the diff, run the tests, merge or send it back with comments.

I didn’t write any of the code in this round. My job was to define the problem, set constraints, pick between approaches, read diffs, run tests, and merge. The GitHub log shows Vonng, Codex, and Claude Code as co-authors — that’s just who did the work.

fix.webp

A few things I noticed about how this runs in practice.

Two heterogeneous agents in opposition catch more than one agent alone. A single agent patching a security bug tends toward confident-sounding fixes that quietly miss a boundary condition. Having a second agent argue against the first filters out most of those.

It forces the tradeoffs into writing. When two implementations diverge, someone has to say why A over B. That exchange is the thing I can actually act on as the person deciding what to merge.

Real maintenance is patch-on-patch, not one-shot. The LDAP STS fix is a good example. The first version landed, and then we realized: successful requests shouldn’t count against the rate limit; X-Forwarded-For shouldn’t be trusted by default; the limiter should key on source IP plus normalized username, not just one. Three follow-up commits before it settled. Iterating through that by hand would have cost a lot more time.


Why this fork exists

Because I use MinIO myself.

MinIO is a production dependency for Pigsty. I need working binaries, a complete console, packages that keep shipping, and someone actually handling CVEs. That keeps the scope narrow. No new features, no turning the repo into a playground. Compatibility, supply chain, fixes when they’re needed.

— Chainguard also ships MinIO container images that track upstream’s post-archive commits, a useful option if you use their images. This fork is a different shape: source tree, RPM/DEB packages, restored console, and doesn’t depend on upstream continuing to push patches somewhere.

The fork is at about 1,300 stars on GitHub and 50,000+ pulls on Docker Hub now. Not remarkable numbers, but enough to tell me I’m not the only one who needed this fork to keep shipping.

credit.webp

If you’re already running OSS MinIO, migration is cheap:

You don’t need to replace anything around it or relearn the API. In most cases, you’re just pointing a compatible binary at the same deployment. If you want a full HA production setup, Pigsty ships one for free.


Something I use broke; I’m fixing it.

What’s different in 2026 is the cost of “I’m fixing it.” With two coding agents and someone to referee between them, the maintenance load of a mid-sized Go codebase is tractable for one person in a way it wasn’t a year or two ago. That’s about it — not a grand theory about open-source resilience, just the current operating point.

If you’re running OSS MinIO, the migration is cheap and the patches are current. If another CVE drops, I’ll still be here.

release.webp

1.2 - MinIO Is Dead, Long Live MinIO

MinIO’s open-source repo has been officially archived. No more maintenance. End of an era — but open source doesn’t die that easily.

I created a MinIO fork, restored the admin console, rebuilt the binary distribution pipeline, and brought it back to life.

If you’re running MinIO, swap minio/minio for pgsty/minio. Everything else stays the same. (CVE fixed, and the console GUI is back)


The Death Certificate

On December 3, 2025, MinIO announced “maintenance mode” on GitHub. I wrote about it in MinIO Is Dead.

On February 12, 2026, MinIO updated the repo status from “maintenance mode” to “no longer maintained”, then officially archived the repository. Read-only. No PRs, no issues, no contributions accepted. A project with 60k stars and over a billion Docker pulls became a digital tombstone.

archived.webp

If December was the clinical death, this February commit was the death certificate.

Today (Feb 14), a widely circulated article titled How MinIO went from open source darling to cautionary tale laid out the full timeline.

mermaid-timeline.webp

Percona founder Peter Zaitsev also raised concerns about open-source infrastructure sustainability on LinkedIn. The consensus in the international community is clear:

MinIO is done.

Looking back at the timeline over the past years, this wasn’t a sudden death. It was a slow, deliberate wind-down:

Date Event Nature
2021-05 Apache 2.0 → AGPL v3 License change
2022-07 Legal action against Nutanix License enforcement
2023-03 Legal action against Weka License enforcement
2025-05 Admin console removed from CE Feature restriction
2025-10 Binary/Docker distribution stopped Supply chain cut
2025-12 Maintenance mode announced End-of-life signal
2026-02 Repo archived, no longer maintained End of project

A company that raised $126M at a billion-dollar valuation spent five years methodically dismantling the open-source ecosystem it built.


But Open Source Endures

Normally this is where the story ends — a collective sigh, and everyone moves on.

But I want to tell a different story. Not an obituary — a resurrection.

MinIO Inc. can archive a repo, but they can’t archive the rights that the AGPL grants to the community.

Ironically, AGPL was MinIO’s own choice. They switched from Apache 2.0 to AGPL to use it as leverage in their disputes with Nutanix and Weka — keeping the “open source” label while adding enforcement teeth. But open-source licenses cut both ways — the same license now guarantees the community’s right to fork.

Once code is released under AGPL, the license is irrevocable. You can set a repo to read-only, but you can’t claw back a granted license. That’s the beauty of open-source licensing by design: a company can abandon a project, but it can’t take the code with it.

So — MinIO is dead, but MinIO can live again.

That said, forking is the easy part. Anyone can click the Fork button. The real question isn’t “can we fork it” but “can someone actually maintain it as a production component?”


Why would I do that?

I didn’t set out to take this on. But after MinIO entered maintenance mode, I waited a couple of weeks for someone in the community to step up.

But I didn’t find one. So I did it myself.

Some background: I maintain Pigsty — a batteries-included PostgreSQL distribution with 460+ extensions, cross-built for 14 Linux distros. I also maintain build pipelines for 290 PG extensions, several PG forks, and dozens of Go Projects (Victoria, Prometheus, etc.) packaging across all major platforms. Adding one more to the pipeline was a piece of cake.

I’m not new to MinIO either. Back in 2018, we ran an internal MinIO fork at TanTan (back when it was still Apache 2.0), managing ~25 PB of data — one of the earliest and largest MinIO deployments in China at the time.

More importantly, MinIO is an optional module in Pigsty. Many users run it as the default backup repository for PostgreSQL in production. We did consider several alternatives, but none were a drop-in replacement for MinIO-based workflows.

minio-docs-en.webp

We use MinIO ourselves, so keeping the supply chain alive was not optional — it had to be done. As early as December 2025, when MinIO announced maintenance mode, I had already built CVE-patched binaries and switched to them.

releases.webp

pgsty/minio RELEASE.2025-12-03T12-00-00Z


What We’ve Done

As of today, three things.

1. Restored the Admin Console

This was the change that frustrated the community the most.

In May 2025, MinIO stripped the full admin console from the community edition, leaving behind a bare-bones object browser. User management, bucket policies, access control, lifecycle management — all gone overnight. Want them back? Pay for the enterprise edition. (~$100,000)

We brought it back.

gui.webp

The ironic part: this didn’t even require reverse engineering. You just revert the minio/console submodule to the previous version. They swapped a dependency version to replace the full console with a stripped-down one. The code was always there.

console.webp

We put it back.

2. Rebuilt Binary Distribution

In October 2025, MinIO stopped distributing pre-built binaries and Docker images, leaving only source code. “Use go install to build it yourself” — that was their answer.

For the vast majority of users, the value of open-source software isn’t just a copy of the source — supply chain stability is what matters. You need a stable artifact you can put in a Dockerfile, an Ansible playbook, or a CI/CD pipeline — not a requirement to install a Go compiler before every deployment.

We rebuilt the distribution:

Docker Images
pgsty/minio is live on Docker Hub. docker pull pgsty/minio and you’re good.
RPM / DEB Packages
Built for major Linux distributions, matching the original package specs.
CI/CD Pipeline
Fully automated build workflows on GitHub, ensuring ongoing supply chain stability.

If you’re using Docker, just swap minio/minio for pgsty/minio.

For native Linux installs, grab RPM/DEB packages from the GitHub Release page. You can also use pig (the PG extension package manager) for easy installation, or configure the pigsty-infra APT/DNF repo to install from it:

curl https://repo.pigsty.io/pig | bash; 
pig repo add infra -u; pig install minio

Just works as usual.

3. Restored Community Edition Docs

MinIO’s official documentation was also at risk — links had started redirecting to their commercial product, AIStor.

We forked minio/docs, fixed broken links, restored removed console documentation, and deployed it as the SILO documentation site.

The docs use the same CC Attribution 4.0 license as the original, with necessary maintenance.

doc.webp

Commitments

Some things worth stating up front to set expectations.

No New Features — Just Supply Chain Continuity

MinIO as an S3-compatible object store is already feature-complete. It’s a finished software. It doesn’t need more bells and whistles — it needs a stable, reliable, continuously available build. (I already have PostgreSQL for these, so I don’t need something like S3 table or S3 vector. A stable S3 core is all I need)

What we’re doing: making sure you can get a working, complete MinIO binary, with the admin console included and CVE fixed. RPM, DEB, Docker images — built automatically via CI/CD, drop-in compatible with your existing minio. We keep the existing minio naming and behavior where legally and technically feasible.

This Is a Production Build, Not an Archive

We run these builds ourselves and have been dogfooding them in production for three months. If something breaks, we detect it early and patch it quickly.

I build this primarily for Pigsty and our own usage, but I hope it helps others too.

I’m willing to Track CVEs and Fix Bugs

If you run into issues, feel free to report them at pgsty/minio. I’ll do my best to fix these — but please don’t treat this as a commercial SLA.

Given that AI coding tools have made bug fixing dramatically cheaper, and that we’re explicitly not adding any new features, I believe the maintenance workload is manageable. (how often do you see one?)

Trademark Is Tricky, But We’ll Cross That Bridge When We Come to It

Disclaimer

Trademark Notice: MinIO® is a registered trademark of MinIO, Inc. This project (pgsty/minio) is an independently maintained community fork under the AGPL license. It has no affiliation with, endorsement by, or connection to MinIO, Inc. Use of “MinIO” in this post refers solely to the open-source software project itself and implies no commercial association.

AGPLv3 gives us clear rights to fork and distribute, but trademark law is a separate domain. We’ve marked this clearly everywhere as an independent community-maintained build.

If MinIO Inc. raises trademark concerns, we’ll cooperate and rename (probably something like silo or stow). Until then, we think descriptive use of the original name in an AGPL fork is reasonable — and renaming all the minio references doesn’t serve users.

AI Changed the Game

You might ask: can one person really maintain this?

It’s 2026. Things are different now.

AI coding tools are changing the economics of open-source maintenance.

With tools like Claude Code & Codex, the cost of locating and fixing bugs in a complex Go project has dropped by more than an order of magnitude. What used to require a dedicated team to maintain a complex infra project can now be handled by one experienced engineer with an AI copilot.

Maintaining a MinIO build without adding new features is a manageable task. The key requirement is testing and validation. and we already have that scenario, which lets us verify compatibility, reliability, and security in practice.

Consider: Elon cut X/Twitter’s engineering team down to ~30 people and the system still runs. Maintaining a MinIO fork without new features is considerably less daunting


Just Fork It

MinIO Inc. can archive a GitHub repo, but they can’t archive the demand behind 60k stars, or the dependency graph behind a billion Docker pulls. That demand doesn’t disappear — it just finds its way out.

HashiCorp’s Terraform got forked into OpenTofu, and it’s doing fine. MinIO’s situation is actually more favorable — AGPL is more permissive for forks than BSL, with no legal gray area for community forks. A company can abandon a project, but open-source licenses are specifically designed so the code can’t die.

Fork is the most powerful spell in open source. When a company decides to shut the door, the community only needs two words:

Fork it.


Reference

Originally published in Chinese

1.3 - MinIO Is Dead. Which Next?

MinIO announced maintenance mode two days ago. I ranted in “MinIO Is Dead” and immediately got flooded with “so what now?”

The usual suspects: Ceph, RustFS, SeaweedFS, Garage. I packaged all of them for Linux (RPM/DEB) and ran them through the grinder.

Short version: there’s no perfect substitute. Ceph is powerful but overkill; SeaweedFS rocks tiny files but needs an external metadata DB; Garage is cute but too barebones; RustFS targets the MinIO niche but is still alpha.

Quick scan of the field

MinIO is the open-source S3 clone. If all you need is basic object CRUD, any S3-compatible store works. But parity with MinIO means more than APIs—it’s about reliability, operability, tooling, documentation, SOPs. Replacing it cleanly is hard.

Ignoring commercial clouds, here’s the OSS menu:

  • Ceph – arguably the best choice for enterprises, but brutally complex. Most folks don’t need block + file + object in one, and it requires extras like Podmon. MinIO’s single binary spoiled us.
  • SeaweedFS – optimized for oceans of small files; O(1) disk seeks make it absurdly fast there. But it relies on an external metadata store. If you want a general-purpose object store, that dependency is annoying.
  • Garage – built by Deuxfleurs with NGI funding. Delightfully light (10 MB), great for self-hosters and edge nodes. But S3 compatibility is thin: no versioning, no cross-region replication, no IAM. Enterprises will laugh.
  • RustFS – the only project explicitly chasing “drop-in MinIO,” but it’s still alpha.

RustFS vs. MinIO

RustFS looked the most promising, so I wired it into Pigsty as a MinIO replacement. Most logic carried over, but a few differences popped up:

  • Certificates must follow specific naming rules.
  • Health checks differ from MinIO’s endpoints.
  • mc admin doesn’t work; you can’t push fine-grained IAM policies. That’s a deal-breaker for many teams.

It ran, but I’m not shipping alpha software into production, so I shelved the branch. I’ll revisit when RustFS hits GA.

Will RustFS repeat MinIO’s mistakes?

RustFS has potential, but I worry it’ll retrace MinIO’s path. I asked the AI big three (GPT‑5 Pro, Claude 4 Opus, Gemini 3 Pro) to audit the project. Gemini leveled some serious accusations; Claude corroborated.

The red flags match MinIO’s history: Apache 2.0 license + copyright assignment CLA + single commercial gatekeeper. With that risk profile, I’m downgrading RustFS from “optimistic” to “cautious wait-and-see.”


So what now?

Pigsty bundles MinIO as an optional module for PostgreSQL backups or as an on-prem S3 for apps like Supabase. After surveying the alternatives, I’m not eager to swap it out. I might add a pgBackRest-native backup server option, but ripping out MinIO today feels premature.

Best plan: stay on the latest MinIO release, lock the version, isolate it on the network, and wait a few months. Maybe the community forks it; maybe RustFS matures. Adjust when reality changes.

RustFS still has a golden window to seize MinIO’s niche with a safer, community-friendly fork. That window is measured in months, not years.


If you stick with MinIO

Use the latest build, not the April 22, 2025 edition with the GUI. There’s a serious CVE in the interim:

  • CVE-2025-62506 – privilege escalation via session-policy bypass (HIGH). Low-privilege users can mint new accounts and escalate.

In a locked-down intranet the risk is manageable, but you still want the fix, which landed in the 2025‑10‑15 release. MinIO pulled the prebuilt binaries starting with that version, offering source only. Annoying, but it’s Go—go build and you’re done. I forked MinIO, ran their packager, and produced RPM/DEBs for 2025‑12‑03 so I’m not deploying vulnerable bits: https://github.com/pgsty/minio

minio.png

Security patches still need humans. MinIO claims they’ll fix critical issues, but if the community wants a maintained fork, now’s the moment. Start from 2025‑04‑22, cherry-pick critical bug/security fixes, and keep a community LTS alive.

MinIO is “done” software. It doesn’t need the latest S3 gimmick (Vector/Table); it needs steady bugfixes. That’s perfect for a community branch. Plenty of storage vendors rely on MinIO; maintaining a fork beats writing a new object store from scratch.

2026-02-14 Update: MinIO’s official repo has been fully archived and is no longer maintained. Besides, I’ve personally maintained an oss fork of minio: pgsty/minio / Docs: https://silo.pgsty.com. Which based on the last upstream version 2025-12-03 with restored console capabilities.

1.4 - MinIO is Dead

December 3, 2025 was a day to mark in open-source software history. MinIO’s team updated the project status on GitHub, announcing the MinIO open-source project was entering “maintenance mode.” This basically declared the death of MinIO as an open-source project.

MinIO the company has finally completed its transformation from a dragon-slaying hero into the very dragon it once sought to slay.

maintenance-mode.png

From Dragon-Slayer to Dragon

Democratization Era (2014–2019): The Apache of Object Storage

MinIO was founded in 2014 with a highly idealistic vision – to be “the Apache of object storage.” In an era dominated by AWS S3, MinIO’s ultra-lightweight design (a single static binary) and 100% S3 API compatibility quickly won developers’ hearts.

During this phase, MinIO was licensed under the liberal Apache 2.0 license, encouraging developers to integrate it into all kinds of applications. Its core pitch: “turn any hardware into AWS S3.” This open strategy was wildly successful. MinIO claimed its Docker image had been pulled over 1 billion times, making it the world’s most widely deployed object storage service. At this point, MinIO was a darling of the cloud-native stack – the default storage backend in many Kubernetes setups.

License Weaponization (2019–2025): The AGPL War

The first major crack in community relations appeared around 2019–2021. MinIO announced it was changing its core license from Apache 2.0 to GNU AGPLv3.

The official explanation was that this move aimed to prevent cloud providers (like AWS, Azure) from “freeloading” the code and repackaging it as proprietary services — a common defensive tactic in open source. During this period, MinIO shifted from being a community guardian to an aggressive defender of its IP. In 2022, MinIO publicly accused Nutanix Objects of violating its license and revoked Nutanix’s right to use MinIO; in 2023, MinIO sued high-performance filesystem vendor Weka on similar grounds. These legal actions, though legally contentious, sent a clear signal: MinIO no longer welcomed commercial use without paying up. This set the legal and psychological stage for the full lockdown that would come in 2025.

Control Plane Neutered (May 2025)

In May 2025, MinIO decided to strip the MinIO Console out of the community edition. The console was a critical GUI for bucket management, IAM, monitoring, and audit logging. After this removal, the open-source MinIO was left with only a basic “object browser” GUI – essentially just a file viewer/downloader.

Meanwhile, key admin features like policy management, site replication configuration, and lifecycle management were moved entirely into the commercial enterprise edition. This change downgraded the open-source MinIO from a full-featured storage management system into a mere data-plane component, robbing it of the control-plane capabilities needed to run as a standalone product in production.

Cutting Off Binary Distribution (Oct 2025)

On October 15, 2025 – right as a critical security vulnerability (CVE-2025-10-15T17-29-55Z / GHSA-jjjj-jwhf-8rgr) was disclosed – MinIO stopped publishing updated Docker images to Docker Hub and Quay.io. The timing of this move was highly strategic. By cutting off binaries during a major security incident, MinIO effectively used security as a bargaining chip.

This decision directly broke the automated deployment pipelines for countless users. Helm charts, Ansible playbooks, and Terraform scripts expecting minio/minio (or Bitnami’s minio) image suddenly failed to find updates. Auto-scaling groups trying to pull new nodes hung due to missing images. For teams without a Go build environment or an internal container registry, MinIO instantly became unusable.

Maintenance Mode (Dec 2025)

On December 3, 2025, MinIO, Inc. officially updated its channels and GitHub repo to announce that the open-source project is now in “maintenance mode.” The README stated that there will be no further feature additions or improvements, issues and PRs will no longer be reviewed, and even critical security fixes would be provided “as appropriate.” No more RPM/DEB packages or Docker images will be released. Essentially, anyone needing updates or support is advised to switch to the commercial AIStor product.

aistor.png

Technical Impact: Damage to the Open-Source Ecosystem

MinIO’s move to maintenance mode dealt an immediate and far-reaching blow to many tech stacks.

Broken CI/CD Pipelines and an Automation Crisis

Thousands of Helm charts, Ansible playbooks, and Terraform scripts depend on the minio/minio (or Bitnami’s minio) container image. With official images no longer published, third-party packagers like Bitnami — who can’t get a stable upstream release — also had to stop updates.

  • Cascade effect: Deployments in fresh environments started failing outright. Auto-scaling groups, upon launching new instances, would hang or error out when the MinIO image couldn’t be pulled.
  • Cost of fixes: Companies now have to rewrite their deployment scripts to point to a self-hosted image, and set up internal build pipelines to compile and package MinIO from source.

Security Vacuum: CVE Patches Go Private

The most lethal consequence of halting binary distribution is delayed security patches. In the October 2025 incident, for example, MinIO effectively withheld the patched binaries for the vulnerability.

  • Risk exposure: Companies without dedicated security teams are forced to keep running older, vulnerable versions with known critical flaws.
  • Compliance nightmare: For organizations under PCI-DSS, HIPAA, SOC2, etc., not being able to obtain vendor-signed security updates is a compliance disaster. Lacking official patches, they technically fall out of compliance.

Exponentially Higher Ops Complexity

Removing the UI wasn’t just a hit to user experience – it increased operational burden. Tasks that used to be a few clicks in the Console (configuring bucket policies, setting user permissions) now require ops engineers to master the mc CLI or hand-craft complex JSON policy docs. This raises the skill floor and makes MinIO far less friendly as a lightweight internal tool.


Underlying Reasons: Pressure from Capital and Commercialization

The driving force behind MinIO’s decisions is the logic of venture capital. By 2025, MinIO had raised a total of $126 million in funding. The most significant was a $103 million Series B in January 2022 led by Intel Capital, SoftBank Vision Fund II, and General Catalyst, which crowned MinIO a unicorn (valued over $1 billion).

In VC terms, a $1B valuation means the company must show a clear path to IPO — typically demanding $100M+ in Annual Recurring Revenue (ARR) and rapid growth. In Feb 2025, MinIO announced its ARR had grown 149% over the past two years businesswire.com. Impressive growth, but to live up to a sky-high valuation, organic conversion alone wasn’t enough.

Cutting off the free open-source offering is the most direct way to force a huge user base into paid customers.

In 2025, MinIO underwent a full rebrand and launched “MinIO AIStor,” styling itself as “the data backbone for enterprise AI.” Management recognized that general-purpose object storage (for backups, file servers, etc.) was a red-ocean market with thin margins, whereas generative AI’s appetite for high-throughput data (the exascale AI era) promised the next big surge. By tuning its product for AI workloads and focusing on Fortune 500 enterprises linkedin.com, MinIO essentially decided to cut loose its low-value open-source user base. The move to maintenance mode signaled MinIO’s official pivot from a broad open-source project into a vertical, high-end AI software vendor.

MinIO isn’t a garage hobby project by a few geeks anymore; it’s a company that took $126M in VC and is valued at over $1B. Backed by Intel Capital and SoftBank, once you take that money, your boss is no longer the users — it’s the investors. And what do investors want? ARR, growth, IPO. You tell them, “We have a billion Docker pulls!” and they’ll ask, “How many dimes did those pulls pay us?”

The reality is brutal. To the VCs, those small businesses and individual devs using free MinIO are low-value assets. They open issues and ask for support — consuming expensive engineer time, bandwidth, and servers — yet will never convert to paying customers. MinIO’s leadership knows their real cash cows are the Fortune 500 firms doing generative AI. The ones training GPT models or running self-driving pipelines need AIStor, ultra-high performance, and 24/7 enterprise SLAs.

So flipping the project into “maintenance mode” is essentially an asset carve-out. MinIO is cutting away the “dead weight” (free users) and concentrating on the milkable “cash cows” (enterprise AI clients). In business strategy this is called focus. To the investors, it’s being responsible. But from the perspective of open source, it’s simply betrayal.


Personal Reflections

I started using MinIO around 2018 (back when it was Apache-licensed). We built a few multi-petabyte object storage clusters for videos, images, backups — probably one of the largest MinIO deployments in China at the time. I wrote deployment/monitoring playbooks for MinIO (still open-sourced in Pigsty).

As an open-source startup founder, I can understand the motivation behind these moves. But as an open-source contributor and user — I also know many folks right now have one phrase in their minds: “I have never seen such shamelessness.”

An open-source license isn’t a shackle, but it is a social contract. Developers contribute code, users contribute testing, feedback, and reputation; together, they make a project successful. MinIO enjoyed a decade of community goodwill and parlayed the bragging rights of “#1 in global downloads” into venture funding. Then it turned around and told the very users who propped it up: “You free-riders, get lost.” This kind of move breaks the fundamental trust that open source is built on.

This “bait and switch” tactic is even more nauseating than a crypto rug pull. A rug pull only takes your money — MinIO is pulling the rug out from under the tech stacks of thousands of companies. Adopting a technology isn’t just picking up a binary; it’s buying into an ecosystem and a design philosophy. They got everyone onboard, let the switching costs pile up sky-high, and then suddenly kicked away the ladder. In fact, as open-source expert Tison thoroughly discussed in his article The Bait-and-Switch Open-Source Strategy, the core issue with this model is deception.

MinIO betrayed the community, so the community may abandon it as well. Alternatives like Garage, SeaweedFS, or the new RustFS are ready to step in.

If I have to sum up my feelings, I’d borrow a line from The Hitchhiker’s Guide to the Galaxy:

—— “So long, and thanks for all the fish.”

2026-02-14 Update: MinIO’s official repo has been fully archived and is no longer maintained. Besides, I’ve personally maintained an oss fork of minio: pgsty/minio / Docs: https://silo.pgsty.com. Which based on the last upstream version 2025-12-03 with restored console capabilities.

2 - Release Notes

Detailed notes for every published SILO release, ordered from newest to oldest.

Each published SILO version has its own page with the release date, major changes, security fixes, dependency updates, and related commits.

2.1 - SILO Console 2.4.1 Released

Restricted shared downloads, password-permission migration, streaming ZIPs, refreshed dependencies and verifiable release artifacts.

SILO Console v2.4.1 was released on September 16, 2026 from 1360e26d976d. It integrates pkg v3.14.1 and mcli 20260916, restricts the anonymous sharing proxy, streams multi-object ZIP downloads and adds signed, traceable release artifacts.

Shared downloads

The fix for #52 restricts the anonymous proxy to object-content GET requests at the configured S3 origin. It rejects redirects, system paths and query-selected operations that are not object downloads. Public objects, presigned links and version-specific downloads continue to work. No new sharing-disable environment variable is needed.

Thanks to Jiri Pejchal (@jiri-pejchal) for reporting the internal-metrics exposure. See the sharing boundary for the supported request forms and reverse-proxy requirements.

Password-policy migration

This patch release includes an authorization change. The Change Password button and session capability use admin:ChangeMyPassword; creating users and resetting another user’s password continue to use admin:CreateUser. With the matching SILO Server, denying CreateUser alone no longer locks the caller’s own password. To retain that restriction, deny both actions in the same statement before upgrading, keeping its original scope and conditions.

The updated built-in readonly permits self-service password changes and no longer overrides a separate CreateUser Allow. Saved policy overrides retain their stored statements. Upgrade Server, Console and pkg together and review the migration and rollback guide.

Downloads and browser recovery

  • Multi-selection ZIP downloads stream to a file writer when the browser supports it, with backpressure, cancellation and duplicate-transfer protection. Other browsers hand the download to their native download manager. Neither path buffers the complete ZIP in JavaScript.
  • Progress uses a percentage only when the response supplies a reliable total. The native fallback identifies the handoff explicitly; cancellation then belongs to the browser’s download manager. Selections over 5 GiB or with an unknown total recommend mcli for long-running transfers.
  • Invalid routes and rendering failures offer recovery without clearing preferences. English and Chinese labels, icon controls, sign-out text and keyboard-accessible tooltips are completed.

Dependencies and embedding

Component Selected version
Shared package github.com/pgsty/silo-pkg/v3 v3.14.1
MC source github.com/pgsty/mc v0.0.0-20260916070421-e952aa78f10a — mcli 20260916
Upstream minio-go v7.3.1-0.20260915093545-32e1f32cb176
JWX / strfmt v3.3.0 / v0.27.2
React Router v7.18.4
Build tools Go 1.27.1, Node 24.21.0, Yarn 4.13.0

The SDK handles S3 errors embedded in HTTP 200 CopyObject responses; JWX fixes custom JSON field-name escaping and strfmt updates hostname validation for Go 1.27. The embedded frontend and third-party credits are regenerated from this dependency graph. go-systemd v22.6.0 and tablewriter v0.0.5 remain intentional compatibility pins.

Go embedders select Console as github.com/pgsty/silo-console v0.0.0-20260916075814-1360e26d976d through the historical github.com/minio/console module replacement. Go does not inherit dependency replacements: copy the explicit MC replacement from the tagged README. The supported administration target is the coordinated SILO stack; upstream MinIO/MC compatibility is best effort.

Packages and verification

The official container image is on Docker Hub, with Linux amd64 and arm64 support:

docker pull docker.io/pgsty/silo-console:v2.4.1

The release contains 44 assets: binaries and bundles, DEB/RPM/APK packages, source, legal notices, SPDX SBOMs, a checksum manifest and its Sigstore bundle. Checksums and the OCI image use keyless Cosign signatures; GitHub provenance records the build workflow and tagged release inputs. The image’s latest tag advances only after the published release passes signature, provenance, SBOM and anonymous-access checks. See the release contract for online and offline verification.

Linux packages retain minio-console.service, console-user and /etc/default/console. The service now uses /var/lib/silo-console for state, /etc/silo-console/certs for certificates, and a 90-second shutdown limit. Before restarting an existing installation, migrate certificates with the documented ownership or retain the old path through CONSOLE_OPTS. The installer preserves old certificates and keys and does not restart the service for you. See the package upgrade instructions.

The exact release source passed the complete CI matrix, vulnerability checks and release verification. Local validation also covered 251 frontend unit tests, 51 browser tests, byte-identical embedded assets and sharing in standalone and embedded deployments. The downloaded Darwin/arm64 executable was independently checked against the signed checksum manifest and exercised against a SILO test server.

2.2 - silo-pkg 3.14.1 Released

JWX JSON encoding hardening and an upstream CopyObject response fix.

v3.14.1 was published from fa657ef4. This is a dependency and tooling release of github.com/pgsty/silo-pkg/v3; the diff from v3.14.0 changes no package Go source or public API.

Dependency changes

  • JWX advances from v3.2.0 to v3.3.0. The vendor’s release notes identify GHSA-4cf7-xm37-g63h: custom JSON field names must be escaped during encoding. Reachability depends on accepting attacker-controlled custom claim names. The package’s env JWT path uses registered claim names; dependency presence alone does not establish an exploitable mcli path.
  • Upstream minio-go advances to v7.3.1-0.20260915093545-32e1f32cb176, including minio-go #2306: recognize an S3 CopyObject error embedded in an HTTP 200 response instead of reporting a successful copy. The fix applies to the SDK’s CopyObject path; it does not establish every application’s selection of that path.
  • Testify advances to v1.12.1 and the lint configuration adopts gomodguard_v2. The Go 1.26 floor, Go 1.27.1 toolchain and go-systemd NetBSD replacement remain.

Component and migration boundary

The v3.14.0 password-policy change remains in force; this patch does not revert it. Preserve the paired admin:CreateUser / admin:ChangeMyPassword Deny where that was the intended restriction, using the migration guide.

mcli 20260916 and Console 2.4.1 select this package. The Server source baseline reviewed on September 16 still selects v3.14.0; publishing a module does not update a compiled Server. The component matrix separates published components from the next Server’s selected dependencies.

Release verification is recorded in the linked release and PR #9. Those records are upstream release evidence, not a new execution of their tests by this documentation update.

2.3 - mcli 20260916 Released

pkg v3.14.1, upstream SDK CopyObject repairs, reliable failure exit codes and verified packages.

Release: RELEASE.2026-09-16T00-00-00Z · Package: 20260916000000.0.0 · Source: e952aa78f10a.

Changes since 20260913

  • Adopt silo-pkg v3.14.1, upstream minio-go v7.3.1-0.20260915093545-32e1f32cb176, and JWX v3.3.0. CopyObject retries and propagates S3 errors embedded in HTTP 200 responses, so a failed copy cannot authorize mv to delete its source. JWX correctly escapes JSON field names.
  • Report mirror permission failures, including unreadable sources, rejected writes and failed local destination removals. Later objects continue; a finite mirror with failures exits 1. These per-object permission errors do not require --skip-errors and do not restart an entire watch scan.
  • Preserve the existing cancellation/retry path for listing and watcher failures. Persistent listing permission denial can still cause repeated scans; this release does not redesign that retry policy.
  • Suppress normal final success statistics after a failed mirror. Explicit --summary retains statistics: JSON uses status: failure; text error diagnostics remain on stderr. Transfers completed within one clock tick no longer produce infinite JSON speeds.
  • Return failure for legalhold set/clear errors, partial recursive retention failures, and moves whose source deletion fails. mv waits for deletion results; successful copies are not rolled back. Retention reports each failed object once instead of duplicate or misleading diagnostics.
  • Reject empty retention durations and invalid find --regex expressions with normal CLI errors instead of panics.

Automation and upgrade notes

Affected failure paths that returned 0 now return 1. Check final exit status and error records. Per-object success start messages and progress byte counters are not proof that a transfer completed. For mv, a deletion failure may leave both the source and the successfully copied destination; inspect both before retrying.

Configuration format, aliases, MC_* variables, commands and package naming remain unchanged. The password-policy split introduced by pkg v3.14.0 is unchanged in v3.14.1. Upgrading only mcli does not update an installed Server or Console, alter Server authorization, or replace a client bundled in an existing Server image. See the component matrix and password-policy migration.

SILO is the maintained administrative integration target. Compatibility with unmodified upstream MinIO and other S3 services remains best effort.

Delivery and validation

The release contains 19 assets: six Linux/macOS/Windows archives for amd64/arm64, six Linux RPM/DEB/APK packages, one archive checksum manifest and six package checksum sidecars. Archives, the manifest and DEB/APK packages have verified Sigstore build attestations bound to this signed tag and source commit. RPMs carry the PGSTY GPG signature (9592A7BC7A682E7333376E09E7935D8DB9BD8B20).

The final release commit passed Go tests on Linux, macOS and Windows with the race detector, Linux 386 tests, cross-compilation, vulnerability scanning and the release packaging pipeline. Pre-release acceptance also exercised both the published SILO 20260903 and fixed Server source 3c26a8b0b5bd, Object Lock and permission failures, CopyObject error injection, and both-architecture RPM/DEB/APK installation and upgrades. No reachable or imported-package vulnerability was found; an unused OpenPGP module-only advisory remains.

CI: VulnCheck · Test Release Pipeline · Crosscompile · Go · Release · Docker

The docker.io/pgsty/mc release and latest tags resolve to the verified Linux amd64/arm64 manifest:

sha256:cfc83108c3abb371f8fb84d99c1fdc88f8c237e022409b0081fb7c0a3be634dd

Download and installation · Client compatibility · Changes since 20260913.

The HTTP-200 CopyObject repair applies when mcli selects a single CopyObject: objects smaller than 64 MiB or explicit --disable-multipart. Larger copies normally use ComposeObject; this fix does not substitute for separate evidence for that path.

See pkg 3.14.1 for the precise JWX/GHSA-4cf7-xm37-g63h scope. A failed --json mirror without --summary emits no trailing statistics record; --skip-errors does not turn recorded failures back into exit 0. Some retention JSON failure records carry status: failure with an error object serialized as {}; consumers must not require a human-readable message.

2.4 - mcli 20260913 Released

Correctness fixes, pkg v3.14.0, upstream SDK refresh and verified multi-architecture delivery.

Release: RELEASE.2026-09-13T00-00-00Z · Package: 20260913000000.0.0 · Source: 4f609a4d.

Changes since 20260903

  • Preserve historical destination versions under mirror --remove --watch.
  • Keep service-restart dry runs from executing a restart and make noninteractive behavior explicit.
  • Return failure for transfer and S3 Select errors, honor explicit checksums on empty uploads, and retain pipe JSON output when quiet mode is enabled.
  • Accept on/off boolean environment values and repair CLI/JSON behavior across supported platforms.
  • Adopt pkg v3.14.0 and upstream SDK v7.3.1-0.20260910142817-60bd07042d49, refresh Go x/* and UBI dependencies, and keep Go 1.27.1. NetBSD retains the go-systemd v22.6.0 portability pin.

The package retains the policy Deny/NotResource and bounded wildcard repairs from pkg v3.13.3. Correctly preserved denies may now reject requests that relied on lost clauses. Recover already-lost clauses from the original policy source.

The new pkg also changes password capabilities. Read the password-policy migration before a coordinated Server upgrade. This mcli release alone does not change an installed Server’s action mapping. September 16 follow-up: the matching password UI is released in Console v2.4.1; see the component matrix for the coordinated upgrade.

Delivery and validation

The immutable release contains 19 assets: six archives for Linux, macOS and Windows on amd64/arm64, RPM/DEB/APK packages for both Linux architectures, checksums and package checksum sidecars. Archives and the checksum manifest have verified build attestations; RPMs carry the PGSTY GPG signature. Native macOS arm64 version and Go build metadata were checked against the released pins.

The release and latest tags of docker.io/pgsty/mc resolve to the same verified amd64/arm64 manifest:

sha256:aa5cc1401b3e1ab482d215d5717e9e69b4f14970a3656f330ed20a549fe19020

Container publication completed; both architecture images were pulled and their version output checked. Go, cross-compilation, vulnerability and Test Release gates passed before tagging. No reachable or imported vulnerable package was reported; unused OpenPGP retains module-only GO-2026-5932. Compatibility with upstream MinIO and other S3 endpoints is best effort; SILO is the maintained administrative integration target.

Download and installation · Compatibility notes · Complete source diff.

pipe streams nonempty stdin until EOF; empty input uses a regular zero-byte PUT instead of unknown-size multipart. Explicit checksum options reach the upload path. Quiet/non-TTY execution keeps progress disabled while retaining JSON results; that output correction is not limited to empty input.

2.5 - silo-pkg 3.14.0 Released

Password-capability migration, upstream SDK fixes and the September 13 dependency refresh.

v3.14.0 was published on 2026-09-13 from 827f8109. It is consumed directly as github.com/pgsty/silo-pkg/v3.

Password-policy migration

This release changes authorization semantics. The public Policy.IsAllowedActions method now reports admin:ChangeMyPassword unless explicitly denied; admin:CreateUser requires an explicit Allow. With matching Server/Console source, a CreateUser deny alone no longer locks the caller’s password, while a ChangeMyPassword deny does.

The built-in readonly drops its CreateUser deny. A separate CreateUser Allow can therefore become effective; the added consolereadonly follows the same split and includes bucket listing. Neither policy grants user administration by itself. Saved policies and overrides are not rewritten.

To preserve the old combined restriction, retain both actions in the same Deny statement, including its original scope and conditions, before upgrading and throughout rollback. Old Servers do not enforce a password-only deny for this endpoint. Read the full migration guide.

September 16 follow-up: Console v2.4.1 ships the matching password UI with pkg v3.14.1 and mcli 20260916. Follow the component matrix for the coordinated upgrade.

Dependencies and verification

  • Upstream minio-go is pinned to v7.3.1-0.20260910142817-60bd07042d49: configurable upload limits, streaming Content-Type signing, caller TLS trust for RDMA, consistent listing checksums and optional restore status.
  • Go x/* dependencies were refreshed, with govulncheck 1.8.0. The library retains its Go 1.26 floor and Go 1.27.1 toolchain; public Go signatures are unchanged.
  • The go-systemd v22.6.0 replacement remains necessary for NetBSD compilation.
  • Full race suites passed on Go 1.26.8 and 1.27.1, along with lint and LDAP configuration validation. Vulnerability scanning found no reachable or imported vulnerable package; unused OpenPGP code retains module-only GO-2026-5932. The release resolves through the Go proxy and checksum database.

The earlier v3.13.3 policy Deny/NotResource and bounded wildcard fixes remain included. Policies whose clauses were already lost must be recovered from their original source. The module-path migration introduced in v3.13.0 is complete in all four maintained components.

Source changes since v3.13.3 · upstream adoption record.

2.6 - SILO Console 2.4.0 Released

Bounded object-browser pages and the accumulated correctness fixes since v2.3.0.

v2.4.0 was published on 2026-09-08 from c103d08e. September 16 follow-up: Console v2.4.1 is the current release. This article records the v2.4.0 behavior and dependencies.

Released behavior

  • Object-browser directories load 100 entries per page by default, with choices of 50, 100, 250, 500 and 1000. Each page uses one S3 continuation-token request. Sorting, filtering and selection apply to the displayed page, not a complete directory scan; there is no unbounded “all” mode.
  • Version deletion stays bound to the selected object and directory deletion to its prefix. Lifecycle edits preserve independent actions and unexposed settings; replication rule deletion is saved consistently.
  • Transient network failures preserve sessions. Empty versioning state, malformed sidebar preferences and stale dropdown selections are handled.
  • Embedded Console restores loopback proxy behavior when Server explicitly enables the policy; standalone trust defaults remain unchanged.
  • Dependencies include pkg v3.13.3, MC source v0.0.0-20260908140805-c8aa5d25a63a, and upstream SDK 0e78d3f18efe. This release consumed MC as source and did not itself publish a new mcli tag.

Correctly preserved Deny/NotResource clauses may reject requests that relied on old deduplication defects. Restore already-lost clauses from the original policy source. Embedders must copy the MC replacement from the tagged README.

Later release

Console v2.4.1 adds restricted shared downloads, the password-permission split, streaming multi-object ZIPs, browser recovery and verifiable release artifacts. It selects pkg v3.14.1 and mcli 20260916. v2.4.0 uses the earlier browser-memory ZIP path. Read the password migration before upgrading.

The exact v2.4.0 source passed the complete Console CI matrix. The authoritative historical record is its tagged changelog, object-browser guide, and release assets. Later signature/provenance requirements do not retroactively apply to old assets.

2.7 - Silo Console 2.2.0 Release Notes

SILO Console 2.2.0 release notes: safe text previews, downloads that cannot lie, repaired permission gating, an untangled user API, hardened notification and IAM policy writes, and a Go 1.27 dependency stack on maintained SILO forks.
Note

Released on 2026-08-26. v2.2.0 points to final commit 7dc4258a6. The exact tagged tree passed the full CI matrix, vulnerability checks, and release pipeline. The public release contains six standalone binaries, nine Linux packages, and a SHA-256 checksum manifest.

Version: v2.2.0 · Release commit: 7dc4258a6 · Status: released · Repository: pgsty/silo-console

SILO Console 2.2.0 is a correctness-and-hardening release. It adds one new feature — a strictly bounded text preview for logs and structured text — and spends the rest of its budget making existing surfaces tell the truth: downloads that cannot silently ship a truncated archive, progress bars that cannot fabricate a percentage, permission gates that actually disable what they claim to disable, a user API that no longer entangles status changes with group membership, and database notification forms that emit exactly the connection string the server will store.

Underneath, the dependency stack moves to Go 1.27.0 and onto the maintained SILO forks: pgsty/silo-pkg 3.12.1 replaces upstream minio/pkg, pgsty/mc replaces upstream mc as a library, minio-go moves to 7.3.0, and the etcd client line moves to 3.7.1, closing CVE-2026-73500.

The final change set since v2.1.1 is 33 commits touching 176 source files (+8,794/−2,057 lines, not counting the regenerated embedded frontend assets).

Why 2.2.0 and not 2.1.2: the user-visible fixes alone would justify a patch release, but this cycle changes the provider of the shared policy/certificate implementation, crosses an etcd client minor boundary, splits a REST route, and deliberately changes download-failure semantics. Each of those deserves a minor-version line in the compatibility notes below rather than a silent patch.

Text Preview for Logs and Structured Text

The object browser can now preview .log, .txt, .json, and .xml objects as literal text. The implementation is deliberately paranoid, because “render arbitrary bucket content in the admin UI” is an XSS invitation:

  • Bounded by construction. The request carries Range: bytes=0-1048576; the response is streamed into a fixed buffer with a hard cap, and Content-Range/Content-Length are strictly validated against what was actually read. An object over 1 MiB reports too large — it is never partially rendered as if complete.
  • Text or nothing. The bytes must decode as strict UTF-8 (TextDecoder with fatal: true) and must not contain NUL; anything else reports not previewable instead of rendering mojibake or binary junk. Eligibility is an exact allowlist — the four extensions plus the precise MIME types text/plain, application/json, application/xml, text/xml — and .html/.htm/.xhtml are explicitly excluded even when their metadata claims text/plain.
  • No active document. Content renders into a single DOM text node — never innerHTML, never an iframe, never a JSON-to-HTML transform. A regression suite feeds it HTML, SVG, and XML payloads and asserts they stay inert text.
  • Race-proof. Switching quickly between objects invalidates in-flight previews by generation, so a slow response for the previous object cannot paint over the current one. Cancellation and retry are first-class states.
  • Anonymous-friendly. Public/anonymous object pages get the same preview through the anonymous request path, without triggering any credentialed API call.

Alongside the new preview type, the existing preview plumbing was corrected: empty objects preview as empty instead of erroring, appended logs re-preview at their new length instead of the stale listed size, and metadata races when flipping between objects are gone. The preview and share dialogs now receive the object’s real size (92e8f4e65).

Downloads That Cannot Lie

The whole download path — progress reporting, single objects, folders, archives, byte ranges — was rebuilt around one principle: a download either completes correctly or fails visibly.

Honest progress

A missing, invalid, or contradictory total no longer becomes a fabricated percentage: the progress indicator stays indeterminate until the real total is known. Zero-byte objects are normalized deliberately instead of falling into the unknown-total path, and abort/cancel are terminal states — a late progress event cannot resurrect a cancelled download. Download requests settle exactly once, JSON error blobs are parsed safely, and generated object URLs are revoked.

Streaming folder downloads

Single-folder downloads now use the browser’s native streaming download path instead of accumulating the entire ZIP in JavaScript memory — a multi-GB folder no longer risks tab death. One consequence of the handoff model: the console’s transfer manager reports the folder download complete once it hands the stream to the browser (a toast says so), and the browser’s own download UI takes over from there — cancelling in the console after handoff does not stop the browser-side transfer. Multi-selection downloads still ride the existing POST response and therefore remain an in-memory Blob; changing that requires a separate API decision and is out of scope here.

ZIP integrity — a deliberate behavior change

Important

This is the one change most likely to be noticed as “downloads broke”. In 2.1.1 and earlier, a folder/ZIP download that failed to read some objects silently skipped them and delivered an HTTP 200 archive missing files. In 2.2.0 any per-object failure — list, stat, read, entry creation, close, or copy — aborts the archive: a clean error if nothing was sent yet, an aborted connection mid-stream, so a truncated ZIP can never pass for a complete one.

The practical consequence: a user whose policy grants List on a prefix but GetObject on only a subset of it — a common IAM setup — previously received a partial archive; now the download fails at the first denied object. That was silent data omission, and 2.2.0 treats it as the bug. Download exactly what you can read, or scope the folder download to a readable prefix.

Byte ranges and status codes

  • Malformed or unsatisfiable non-empty Range headers now return 416 with Content-Range: bytes */N instead of a 500. Range parsing is stricter than Go’s lenient default: signs, embedded whitespace, bytes=-0, and empty range elements are rejected.
  • A range request against a zero-byte object returns an empty 200 instead of a 500 — this was the empty-object preview bug.
  • Failures from the lazy object Stat now surface the real S3 status (403, 404, …) instead of a blanket 500.
  • 206 responses now set Content-Length before the header flush, so partial responses carry correct framing.
  • Object sizes are now always serialized — REST and WebSocket listings report "size": 0 for zero-byte objects instead of omitting the field, and the UI displays 0 B. This is the ground truth the honest-progress work stands on.

Version history

S3’s valid null version ID stays visible instead of being dropped, version counting filters prefix matches to the exact object, and history is retained after bucket versioning is suspended or disabled.

Permission Gates That Actually Gate

Row actions were never disabled

Every screen with a data table passed its permission predicate through a prop name (disableButtonFunction) that the table component no longer reads — so view/edit/delete buttons rendered enabled regardless of permission, on every one of the 10 affected screens (Users, Groups, Policies, IDP, webhook settings, bucket access/replication/lifecycle panels). The predicates are now wired to the prop the component actually honors, and a source-guard test fails the build if the dead prop name ever reappears. Server-side authorization was never affected — the buttons produced errors when clicked — but the UI now communicates permissions instead of lying about them.

Independent service-account capabilities

Access-key management previously keyed several UI decisions off one combined permission check. 2.2.0 derives four independent capabilities — List, Create, Update, Remove — in one module and applies them consistently across the self-service Account screen and the admin user-details screen:

  • the service-accounts tab shows when you can list, the create button when you can create, row selection and bulk delete only when you can remove, and the edit pencil only when you can update;
  • View is now genuinely read-only: all fields disabled, no submit path, Enter is inert — previously “view” opened the editable dialog;
  • the session’s advertised Create Access Key capability is computed correctly for request-scoped policies: a Deny conditioned on svc:DurationSeconds (an expiry restriction that can only be evaluated once a concrete request exists) keeps the capability visible — including with wildcard admin actions — while unconditional or login-time denies now properly hide it. The server remains the final authority at request time. The previous code kept the capability visible for any conditional deny, which over-advertised it.
  • an OIDC create/list/get/delete integration regression now runs through the UI’s explicit-credential endpoint, including the expected self-update denial.

Validate IAM policies before writes

Named-policy and service-account policy writes now reject malformed documents and bare S3 resource ARNs with a client error before making an Admin API request. Historical policy reads stay permissive for compatibility, but an incompatible stored policy must be corrected before it can be saved again. The strict parser and resource checks live in the console’s write path instead of importing fork-only policy APIs, preserving the advertised upstream minio/pkg v3.6.1 source-build floor.

Anonymous pages behave anonymously

Anonymous object-browser pages no longer issue protected Object Lock/retention requests that could only produce Access Denied noise, and they gained the language (文/A) and dark-mode controls.

User Status and Groups Are Separate Operations

Toggling a user’s enabled/disabled status and editing their group membership were one combined PUT /user/{name} call that required both payloads and — through the combined permission gate — demanded admin:EnableUser merely to edit groups. 2.2.0 splits them:

  • PUT /user/{name}/status (new) changes only the status, validates the value against enabled|disabled, and rejects the signed-in user’s attempt to enable/disable themselves. Failed toggles no longer leave the switch out of sync — the UI reflects the server’s answer, not an optimistic flip.
  • PUT /user/{name}/groups (existing) is now the only thing group editing calls, and no longer requires admin:EnableUser.
  • PUT /user/{name} survives unchanged on the wire as a deprecated compatibility endpoint for existing API consumers; an unknown status there is now a 400 instead of a 500.

See Compatibility for the API-contract details, including a caveat for regenerated Swagger clients.

Database Notification Forms That Emit What They Mean

The PostgreSQL and MySQL event-destination forms were rewritten around a shared DSN parser/serializer:

  • Structured fields and the raw connection string are one state: the raw string stays authoritative until a structured field is edited, at which point a canonical DSN is rebuilt — libpq keyword/value quoting for PostgreSQL, IPv6-bracket-aware go-sql-driver format for MySQL. Mode switches no longer mangle manually entered strings.
  • Generated previews mask credentials; the mask can never reach the API payload (the payload always uses the raw connection value).
  • Saving requires both a connection string and a table, and cleared values actually propagate.
  • The server now rejects DSNs its own configuration grammar would corrupt — embedded newlines, values that would parse as sibling config keys (table=… inside a password), unbalanced quoting — with a 400 before anything is stored, and without echoing the submitted secret back.
  • Generic password and token fields across notification targets (Kafka, Redis, MQTT, NATS, webhooks) render as password inputs, including environment-overridden values.

Restart honesty

Configuration add, update, delete, and reset now honor the server’s actual restart-required answer instead of hardcoding “restart needed” (or worse, dropping it). The pending-restart flag is monotonic: once any operation requires a restart, later operations that don’t cannot clear it — only an actual service restart does.

Upload advisory

Browser uploads larger than 5 GiB get a non-blocking warning: the console uploads as one non-resumable request, and mcli with multipart upload is the right tool at that size.

Sessions, Metrics Auth, and i18n Hardening

  • Sessions fail fast. An anonymous/empty session gets an immediate 401 instead of hanging on an empty-credential admin request. The console accepts the canonical 401 and the legacy 403 invalid-session responses, and expired-session redirects are subpath-aware — a console served under /console/ redirects within its base path.
  • Prometheus Basic auth works everywhere. The health check and the root-fallback probe now send Basic credentials (previously Bearer-only, so a Basic-auth Prometheus disabled every dashboard widget), Bearer tokens keep precedence, and every response body is drained so keep-alive connections are actually reused.
  • Placeholder substitution is escape-proof by construction. Translated placeholder filling uses a one-pass literal formatter: values containing $&, $1, $`, backticks, or braces stay literal, repeated placeholders all fill, and an AST source-guard test bans the unsafe String.replace pattern from ever returning.

Toolchain and Dependencies

Go 1.27 baseline

Component 2.1.1 2.2.0
go directive, build image, CI matrix 1.26.5 / 1.26.x 1.27.0 / 1.27.x
golang.org/x/crypto v0.54.0 v0.55.0
golang.org/x/net v0.57.0 v0.58.0
golang.org/x/text v0.40.0 v0.41.0
golang.org/x/mod v0.37.0 v0.40.0 (closes CVE-2026-56864/-56865)
golang.org/x/tools v0.47.0 v0.49.0

The SILO forks

require github.com/minio/pkg/v3 v3.6.1

replace github.com/minio/pkg/v3 => github.com/pgsty/silo-pkg/v3 v3.12.1
replace github.com/minio/mc => github.com/pgsty/mc v0.0.0-20260806055018-b0021fd01ccb
  • silo-pkg 3.12.1 is the release dependency. pgsty/silo-pkg v3.12.1 was published on 2026-08-25 and the console pins it directly. It builds on the 3.12.0 release, which carries the policy resource-boundary hardening, condition-key lookup repair, LDAP and certificate-watcher fixes, and the Go 1.27 / etcd 3.7 baseline.
  • The mc library moved to the pgsty/mc fork at a date-tagged pseudo-version. Import paths are unchanged.
  • The require line stays on upstream v3.6.1 deliberately. Go ignores replace directives in dependency modules, so a downstream module that requires this console resolves upstream minio/pkg — and the SILO v3.12.1 tag does not exist upstream. Requiring a real upstream tag keeps the console resolvable downstream; the replace applies the fork for the console’s own builds. The final CI verifies the public source-build surface against upstream v3.6.1. This is compile compatibility: a downstream build without its own top-level replace gets upstream behavior, not the fork’s SILO-specific IAM semantics.
  • go-systemd is pinned back to v22.6.0: v22.7.0 uses CLOCK_MONOTONIC on NetBSD, which doesn’t compile there; the pin holds until upstream ships the fix.

etcd 3.7.1 — client libraries only

All three etcd Go modules move together from 3.6.8 to 3.7.1, closing the TLS-listener denial of service GO-2026-6107 / CVE-2026-73500. etcd 3.7 removes legacy protobuf remnants and makes clientv3.New non-blocking — migrations that matter to consumers that construct clients or embed servers. The console does neither: its only etcd path is silo-pkg/quick operating on an already-created client with plain v3 Get/Put. This upgrades compiled client libraries only — it does not touch an operator’s etcd servers, cluster data, or deployment topology; a server upgrade to 3.7 still follows etcd’s own one-minor-at-a-time procedure.

Third-party maintenance stays conservative

The final release updates minio-go to v7.3.0 after a separate compatibility review, migrates its INI import path, and adds lifecycle-filter XML coverage. A console-side compatibility decoder accepts legacy AccountInfo tag payloads returned by older servers. Other accepted maintenance moves include jwx v2→v3 with httprc v3, go-openapi/swag/conv+typeutils 0.28.0, grpc-gateway 2.29.0, cheggaaa/pb 1.0.30, and go.yaml.in/yaml/v3 3.0.5. Larger unrelated go-openapi, pb/v3, compression, and test-library updates remain deferred.

On the frontend, the vulnerability workflow now covers pushes, manual runs, and development dependencies with immutable installs; vulnerable transitive resolutions were refreshed (fast-xml-parser 5.11, nanoid 3.3.18, @babel/core 7.29.7), dead exports and the unused http-status-codes dependency were removed. The new fast-xml-parser 5.x transitive tree (@nodable/entities, is-unsafe, anynum, fast-xml-builder, path-expression-matcher, xml-naming) was supply-chain-checked during this review: all six packages are published by the fast-xml-parser author’s own account and organization, their installed code is free of execution/network/exfiltration patterns, and the whole tree is development-only — nothing ships in the browser bundle.

Security Review

  • govulncheck at the release tree: zero vulnerable symbols reached, zero vulnerable imported packages. The Swagger build tool scans clean separately.
  • Closed by dependency moves: CVE-2026-73500 (etcd TLS listener DoS), CVE-2026-56864 / CVE-2026-56865 (x/mod verification).
  • Still reported, still unreachable: the module-level GO-2026-5932 openpgp advisory — the console does not import x/crypto/openpgp, and no fixed release exists.
  • The text preview was reviewed as an XSS surface (see above); its regression suite includes active-payload tests.
  • The permission-gating fixes are UI-truthfulness fixes: server-side authorization was never bypassed in 2.1.1; the console simply displayed controls it shouldn’t have.

Compatibility

Nothing changes for deployment plumbing: no environment variable, configuration format, command, binary name, systemd unit, port, or embedded data layout changes. The release binary remains self-contained; Go 1.27.0 is a build-time requirement only.

HTTP API contract (console’s own REST API):

Route Change
PUT /user/{name} Unchanged on the wire; now deprecated. OperationId renamed UpdateUserInfoUpdateUserInfoLegacy, body model renamed to legacyUpdateUser (identical schema). Unknown status: 500 → 400.
PUT /user/{name}/status New. Status-only body (enabled/disabled enum, 422 on violation), returns a user object populated with access key and status only — clients must not read group data from it.
updateUser model Now status-only with enum — breaking for generated-spec consumers; the old shape lives on as legacyUpdateUser.
Object download Error semantics changed: bad ranges 500→416 (+Content-Range: bytes */N), range-on-empty 500→200, Stat failures 500→real S3 status, ZIP failures silent-partial-200→visible failure, 206 responses carry Content-Length.
Listings & WebSocket size always serialized, including 0. Additive.
PUT /configs New 400 class for database DSNs the server’s config grammar would corrupt.
Config reset/delete restart in the response now reflects the server’s real answer instead of always true.
GET /session 401 for empty-credential principals; the advertised Create-Access-Key capability is computed more strictly (see service accounts).
Note

If you generate client SDKs from swagger.yml: the UpdateUserInfo operation now points at /user/{name}/status with a status-only body. Code calling the generated UpdateUserInfo symbol keeps compiling but targets the new route; the old combined call is UpdateUserInfoLegacy. Audit call sites when you regenerate.

Behavior changes an operator may notice:

  1. Folder/ZIP downloads over partially readable prefixes fail instead of silently omitting unreadable objects (details).
  2. Range parsing is stricter than Go’s lenient default; degenerate range headers (bytes=-0, empty elements) now get 416 instead of best-effort handling.
  3. Database notification configs that only worked by accident (DSNs the config grammar mangled on the way in) are now rejected up front with a 400.
  4. The Create-Access-Key control is hidden for sessions whose policy unconditionally denies it (previously any conditional deny kept it visible).
  5. A pending restart-required indicator persists until an actual restart, instead of being clearable by a later unrelated config change.
  6. If a reverse proxy in front of the console compresses /api/v1/…/download responses, the text preview’s strict Content-Length verification will reject every preview as an error. The console itself only compresses static assets — leave API responses uncompressed at the proxy.

Regression Review

Because this release rewrites the download path and re-platforms the shared policy library, the full diff against v2.1.1 was re-reviewed adversarially in five parallel passes (Go API; object browser preview/download; permission gating and service accounts; forms/i18n/session; dependencies/build/CI), each hunting specifically for behavior that worked in 2.1.1 and silently changed.

Verdict: no unintentional regressions found. Every confirmed behavioral difference is one of the deliberate changes documented above. The review did surface two small pre-existing UI-guard defects, now visible because the dead disable-prop was brought back to life with predicates whose argument type was never right:

  • the “cannot delete the Default IDP configuration” row guard compares the row object against the string "Default" and therefore never engages (IDPConfigurations.tsx);
  • the “cannot delete an env-override webhook endpoint” row guard has the same object-vs-string mismatch and is inert (WebhookSettings.tsx).

Neither is a 2.2.0 regression — both predicates were entirely dead in 2.1.1 — and in both cases the server still enforces the real rules. They remain documented follow-up items after the release. Two sharp edges are recorded as known limitations rather than defects: the new PostgreSQL DSN parser accepts only canonical key=value syntax when populating structured fields (an unusual-but-libpq-valid DSN shows empty structured fields, and editing a structured field then rebuilds the DSN from those fields), and TestCafe/Playwright coverage asserts UI gating while live-server deny-path coverage remains the integration suites’ job.

Verification

The release decision combines local release-preparation evidence with remote gates run against the exact tagged tree, 7dc4258a6:

Local release-preparation gatesgo build ./..., go vet ./... (plus -tags testrunmain), gofmt, golangci-lint, go test -race ./..., go tool swagger validate, govulncheck, TypeScript tsc, Playwright, Prettier, knip, release-tag cross-compiles for linux/amd64 and linux/arm64, and go mod verify.

Embedded-assets determinism — the frontend was rebuilt from source through the full pipeline (yarn build + embed optimization) and the result compared against the committed web-app/build: byte-identical, zero dirty files. The nine commits after the earlier 19047161f candidate changed Go compatibility, workflows, and browser-test timing, but not product frontend source or embedded assets.

CI, exact final treeWorkflow run 32888892876 reported 32 successful jobs and one explicitly disabled React-test placeholder. It covered lint, semgrep, Go and API tests, five cross-compile targets, Swagger drift, latest-MinIO source builds, distributed integration, site replication, hermetic SSO, the complete TestCafe permissions matrix, subpath-nginx, Playwright, and coverage. Vulnerability Check 32888899120 passed both jobs on the same commit.

Release pipelinegoreleaser run 32916237254 passed both jobs against v2.2.0 and published the public GitHub release.

Downstream contract — in a scratch tree with every replace removed, go mod tidy + go build ./... succeed against upstream minio/pkg v3.6.1, proving the fork replacement never leaks fork-only symbols into the public module surface.

Test-infrastructure repairs shipped in this cycle (so the gates above actually gate): the Docker-backed integration/replication/SSO suites are back behind the testrunmain build tag (a bare go test ./... no longer tries to start containers); the SSO gate is hermetic — no sudo /etc/hosts edits, no ad-hoc pip installs, pinned SILO image, own port, real teardown; the integration gate asserts the new 416 range semantics and stopped binding its PostgreSQL fixture to a host port; the browser gates run off-Linux by publishing fixture ports; a stale pre-fork “MinIO administrator” selector was fixed; state-mutating TestCafe suites are serialized; Playwright CI installs the committed lockfile immutably.

Release Artifacts

The v2.2.0 GitHub release publishes:

  1. six standalone binaries: Linux amd64/arm64/armv6, macOS amd64/arm64, and Windows amd64;
  2. nine Linux packages: DEB, RPM, and APK for amd64, arm64, and armv6;
  3. silo-console_2.2.0_checksums.txt, plus the SHA-256 digest GitHub records for every asset.

Those assets, the tag, and the release page are verified here. This page does not claim a separately distributed container image or detached signatures.

  • 16960f7ab — fix: keep unknown downloads indeterminate
  • 5968bb37d — chore(deps): align the SILO Go dependency stack
  • 288ab1240 — fix: harden sessions, metrics, and translations
  • ecf3bb492 — fix: harden object previews and downloads
  • 902d9650d — chore: tighten dependency and test gates
  • 194c70c7a — build: prepare SILO Console v2.2.0
  • 927b44e26 — fix: harden database notification forms
  • da2191be9 — fix: clarify console upload and secret limits
  • 6141c2445 — build: refresh SILO Console v2.2.0 assets
  • 097e76155 — chore(deps): bump the shared package fork to v3.12.0
  • 99ca523d6 — fix: split user status updates out of the combined user route
  • f4097992f — fix: honor the server restart result for configuration changes
  • f1280032a — fix: restore permission-gated table row actions
  • 24ce0af97 — fix: keep request-scoped access key conditions visible in the console
  • 92e8f4e65 — fix: pass the object size into the preview and share dialogs
  • 8f6fb3c78 — test: make the SSO gate hermetic and pin it to a SILO release
  • 384a2cb95 — build: refresh SILO Console assets and record the changes
  • a73cda376 — test: fix the integration gate’s stale range and host port
  • cf5049c1d — test: make the browser gates runnable and fix a stale selector
  • 6fa19d857 — fix: complete service account permission boundaries
  • 7e57771a4 — build: refresh SILO Console assets
  • 57cfe7aa0 — fix: restore downstream and browser release gates
  • 19047161f — test: stabilize permissions browser gates
  • e37dec873 — fix: validate IAM policies before writes
  • 28505ed23 — chore: update minio-go to v7.3.0
  • 16abb971e — ci: harden validation and release gates
  • 2ddfcd036 — fix: accept legacy AccountInfo tag payloads
  • 31332bca9 — fix: preserve policy source compatibility
  • 3a8251086 — ci: allow permission tests to finish
  • c159fff78 — ci: serialize shared-role permission tests
  • 2e91cdf9a — test: wait for watch controls to become ready
  • 7dc4258a6 — test: allow asynchronous UI controls to settle

Links:

2.8 - Silo Console 2.1.0 Released

A bilingual console: a zero-dependency English/Chinese interface across every screen, the dashboard migrated to MinIO Metrics V3 with explicit zero-state semantics, and a batch of correctness fixes including escape-proof placeholders and a select-all that cannot lie.

Published: 2026-08-06 · Version: v2.1.0 · Repository: pgsty/silo-console

SILO Console 2.1.0 is the first feature release after the independent 2.0.0. It does three things:

  1. Speaks two languages — every console screen, help topic, and documentation link now renders in English or Chinese, behind a toggle on every page, with zero new runtime dependencies;
  2. Reads the right metrics — the dashboard moves off the MinIO Metrics V2 names onto V3, with explicit handling for the semantics V3 changed underneath it;
  3. Stops lying in edge cases — a select-all that matched what a bulk action would delete, placeholders that survive object names containing $&, timestamps that carry a timezone, and empty metrics that read “no data” instead of a fabricated 0.

This is a minor release. No environment variable, module path, API contract, binary name, or data layout changes. Upgrading is a binary or image swap.

Note

A 2.1.1 patch follows this release

v2.1.1, published the same day, completes the legend hardening described below: a label placeholder the legend builder cannot resolve is now removed instead of leaking literal braces into the Traffic chart legends, the one remaining substitution branch is escape-proofed against label values containing $& or $1, and the License page reports the actual release version instead of 2.0.0. Nothing else changes — upgrade straight to 2.1.1, and everything in this note applies unchanged.

Note

Rebuild your embedded assets if you vendor this console

2.1.0 fixes a packaging defect present on the main branch after 2.0.0: the go:embed payload still carried the 2.0.0 frontend build, so a binary built from an intermediate commit would serve the old UI. The released 2.1.0 artifacts are built from the regenerated payload and are unaffected.

A Bilingual Console

The console is an administration surface for an object store, and a large share of its operators read Chinese first. 2.1.0 makes the interface bilingual without importing an i18n framework — the embedded delivery model means every kilobyte is paid for in the binary. This is issue #6, which proposed i18next; the dependency-free substitution is the one deliberate deviation from it.

How it works

The design constraint was: no new dependency, no build step, no extraction pipeline, and partial coverage must never break the page.

  • English source strings are the dictionary keys. t("Create Bucket") looks up the Chinese entry; a missing key returns the English string unchanged. Coverage can therefore grow incrementally, and a typo degrades to English rather than to a raw key like console.bucket.create.
  • Three dictionaries, one merge. zh.ts (165 chrome entries), zhHelp.ts (247 help-topic entries), and zhScreens.ts (1,373 screen entries) merge with chrome taking precedence — about 1,785 entries in total.
  • The language preference mirrors dark mode: localStoragesystemSlicesetLanguage. There is no browser-locale detection; the default is English, and the choice is explicit.
  • Central interception points rather than per-callsite edits: the page-header wrapper, confirm dialogs, help items, route definitions, and the dashboard’s panel renderer each translate on the way out. This is why 220 screen files could be localized without touching their business logic.
  • Module split matters. i18n/lang.ts holds pure primitives (translate, localizeUrl) and imports no store — systemSlice depends on it, so importing the store back would form a cycle. The hooks (useT, useLanguage, useLocalizedLink) and interpolate() live in i18n/index.tsx.

The toggle is a stroke-drawn 文/A icon mounted in the page header on every page and reused on the login page.

What it covers

Login and SSO flows, navigation and the command palette, the dashboard and every metrics panel, buckets and the full object browser (uploads, previews, sharing, versioning, rewind), users/groups/policies/access keys, configuration and event destinations, IDP and KMS, logs, health reports, speedtest, profiling, inspect, trace, watch, and the license page.

Beyond visible strings:

  • Documentation links localize. silo.pgsty.com links gain a /zh prefix in Chinese; the Pigsty site swaps domains (pigsty.iopigsty.cc). GitHub, MinIO, AWS, and YouTube links are left alone.
  • The help blog feed is per-language, fetching /zh/blog/index.xml in Chinese, with an independent cache per language.
  • The command palette stays searchable in both languages. Menu entries translate for display but keep their English originals as keywords, so “桶” and “buckets” both match.
  • Chart legends translate only their static prefix. translateLegend preserves instance suffixes like [server:drive], and the data layer keeps raw legends so components that match on them for arithmetic (capacity summing) keep working.
  • Timestamps are unified, not merely translated — see below.

What it costs

Roughly +61 KB on the embedded payload (2.79 MB → 2.85 MB, +2.2%), zero new dependencies, and the dictionaries land in their own lazily-loaded chunk. The English rendering path is byte-stable: with the default language, output is identical to 2.0.0.

What stays English

Backend error strings (182 of them) are produced by the Go server and are not translatable from the frontend. A handful of strings hardcoded inside the vendored mds component library — the collapsed-menu “Sign Out” tooltip, and the data table’s “Columns”, “Loading…”, and ON/OFF toggles — remain English; two of them (“Sign Out”, “Actions:”) are swapped via a scoped CSS rule, but the rest would require patching the vendor.

Metrics V3 Migration

The dashboard queried MinIO Metrics V2 names. SILO deployments scrape V3 (/minio/metrics/v3), so the dashboard depended on an endpoint the monitoring pipeline no longer collected. 2.1.0 rewrites all 26 widgets onto the V3 catalog — 31 queries over 29 distinct metric names — and drops three widgets (51/61/62) that no layout ever referenced. This is issue #7; the Info-page half is #8.

The decision is V3-only: no runtime fallback, no probing, no version-selection knob. SILO Console targets SILO deployments, where the server, the scrape pipeline, and the console ship together. The SILO server keeps serving V2 endpoints for external consumers; the console simply stopped using them. A fallback would have been actively harmful — a metrics store retaining 15 days of V2 series would let an or-fallback silently read stale data.

The semantics V3 changed

Three properties of V3 break a naive name-for-name rewrite, and each needed a deliberate answer:

  1. Cluster groups are exported identically by every node. /cluster/* metrics carry no server label and are not leader-gated, so an N-node scrape yields N duplicate series. Queries aggregate with max()/min() — never sum(), which would multiply cluster totals by the node count.
  2. Zero values are not exported at all. Any metric whose value is ≤ 0 is skipped. Offline drive counts, healing-drive counts, and erasure-set health simply vanish rather than reporting 0, which a stat card renders as an empty panel. Every affected query carries a companion guard so the panel reads a real 0.
  3. There is no minio_heal_* namespace. The V2 heal activity signal was in-memory anyway — it reset on restart and bumped on any scan. It is replaced by two cards with defensible semantics: Erasure Health (baselined on write quorum) and Usage Data Age (how stale the scanner’s usage snapshot is).

Zero-state semantics

An adversarial review of the migration produced eight findings, all fixed before release. They share one theme — the difference between zero, no data, and not yet scanned:

  • Capacity free/used baselines on the always-present total, so a full cluster reads 0 free instead of vanishing.
  • Online Drives is guarded against the all-offline case, where the zero-skip would erase the panel exactly when it matters most.
  • Bucket and object counts guard on the usage group’s own freshness gauge, so a cluster that has not completed its first scan reads no data rather than a fabricated 0.
  • Empty single-value results render as , not 0.
  • An empty size distribution no longer fabricates seven zero-height bins.
  • Fractional rates stay visible (parseFloat axis domain, two-decimal CPU formatter) instead of collapsing to 0.
  • Sub-second Usage Data Age clamps to “1 second” instead of rendering blank.

A regression suite (api/admin_info_metrics_test.go) now pins every widget query to the V3 catalog, asserts widget-ID uniqueness, and enforces the per-widget guard taxonomy: health and traffic widgets need a nodes-online companion, usage counts need the usage-group freshness companion, and capacity needs the total baseline. The full mapping is documented in docs/metrics-v3.md.

Also fixed

  • Widget 17 queried sent_bytes twice and widget 11 queried syscall_read twice — both internode/syscall pairs were transposed into duplicates.
  • Label-less matrices (the result of max() aggregation) serialize with no metric field at all, which crashed the frontend’s label extraction and produced a 0 B capacity donut and an empty usage-growth chart. Guarded.
  • An unused per-widget Prometheus label-values prefetch stalled every widget request by up to a second. Deleted.
  • The dashboard’s usage cards, chart controls, and dense Traffic/Resources panels were rebuilt on one grammar and now reflow through tablet widths.

Two server-side bugs were identified during this work and are tracked upstream rather than worked around here: minio_cluster_usage_buckets_since_last_update_seconds emits nanoseconds (the objects variant is correct), and V3 bucket-level sent/received traffic are transposed.

Correctness Fixes

Placeholders that survive real object names

String.prototype.replace interprets $&, $', $`, and $1 in the replacement value as directives. S3 keys legally contain $. So an object named report$&.csv did not render as itself — it re-injected the matched placeholder text into the output and corrupted the message. All 37 dictionary placeholder substitutions now pass the value through a function replacement, where no such interpretation happens. This was a latent bug in the original English UI, not something i18n introduced; the i18n audit is simply what found it.

A select-all that means what it shows

The vendored data table renders a plain untranslatable “Select” header whenever onSelectAll is absent — which was the case on all seven selectable tables. Worse, the naive fix is wrong: a select-all that replaces the whole selection drops rows hidden by an active filter, so the header checkbox and a subsequent bulk action can target different sets. The implementation toggles only the currently visible rows and preserves filter-hidden selections, so the header state can no longer imply a different set than the action would touch.

Timestamps with a timezone

Bucket, object, version, rewind, and access-key timestamps rendered as a mix of verbose English forms and — in several places — a 12-hour clock without AM/PM, which is simply ambiguous. All of them now render as yyyy-MM-dd HH:mm[:ss] (ZZZZ) in both languages.

A translation runtime that survives live data

t() also receives runtime strings: user agents, RSS titles, object names. Two hardening changes followed:

  • misses return unchanged, unconditionally — the implicit @context suffix stripping is gone, because it silently mutated live data that happened to contain @;
  • dictionary lookups are guarded with hasOwnProperty, so a hostile input naming an inherited Object.prototype member (constructor, toString) cannot leak a function into the UI.

Interaction and accessibility

  • An expired session opening a deep link bounced through /login and back, accumulating a redirect chain instead of landing on the form once (#1).
  • Collapsed sidebar buttons carried no accessible name; screen readers announced them as unlabelled (#4). Access Key inputs now declare their autocomplete intent instead of letting password managers guess (#5).
  • Mobile metrics and bucket panels scroll instead of clipping (#3).
  • The speedtest control row wraps instead of overflowing its card, its duration accepts seconds or minutes, and its size defaults to MiB to match its own unit list.
  • Sidebar bucket rows use a virtual row pitch matching the 44px item, so selected and hovered highlights no longer overlap.
  • Unit chips render the selected unit’s label rather than its raw value.

No SUBNET, No Telemetry

Upstream removed Subnet, Registration, and Call Home; this fork inherited that state but still carried three traces. 2.1.0 removes them:

  • the health websocket’s subnetResponse field never addressed a subnet — it is a sentinel meaning “the report was assembled” — and is now reportStatus: "ok";
  • two help topics claimed the health report “uploads automatically to SUBNET” and that inspect output is “transmitted to SILO SUBNET”. Neither was true. They now describe what happens: the report is generated on the deployment and downloaded by the browser;
  • the unreferenced CONSOLE_SUBNET_PROXY constant is deleted.

For the record, 2.1.0’s outbound network posture is unchanged and remains: no analytics, no telemetry, no beacons, no external scripts or fonts. silo-console update is still disabled. The release catalog is contacted only if SILO_RELEASE_SERVICE_HOST (or RELEASE_SERVICE_HOST) is explicitly set — there is no default. The only automatic outbound request the browser makes is the help panel’s blog feed, and only after a user opens the Blog tab.

Upgrade Guide

There is nothing to migrate. No environment variable, module path, protocol field, systemd unit, binary name, or data layout changes between 2.0.0 and 2.1.0.

install -m 0755 silo-console-linux-amd64 /usr/local/bin/silo-console

Two things are worth knowing:

  • The dashboard now requires Metrics V3. If your Prometheus scrapes only the V2 endpoints, dashboard panels will read no-data. Point the scrape at /minio/metrics/v3; Pigsty-managed deployments already do.
  • The language default is English, chosen per browser and stored in localStorage. There is no server-side default and no browser-locale detection, so no existing deployment changes appearance on upgrade.

Verification Scope

Before tagging, the full change set was reviewed and the following gates were run against the final tree: go build, go vet, golangci-lint (0 issues), the Go unit suite across all packages, gofmt, TypeScript type checking, the frontend production build, Prettier across all sources, dictionary duplicate-key checks, and a debug-leftover scan of the complete diff.

The 29 intermediate commits were restructured into 20 logical ones by pure tree operations, and the rebuilt tip was verified byte-identical to the pre-rewrite tree. The embedded payload was rebuilt twice from a clean directory and confirmed byte-identical, which is the property the release pipeline’s zero-diff gate depends on. The pre-rewrite history is retained in a backup ref.

The Metrics V3 migration was additionally reviewed adversarially by an independent model, and all eight findings were fixed (see Zero-state semantics); its queries were validated against a live metrics store with real cluster data.

Known Limitations

  • The SSO end-to-end suite requires an external OpenLDAP/Dex/MinIO topology and was not run in that environment this cycle; the OIDC code paths are covered by unit tests.
  • Backend error strings and several vendored mds component strings remain English (see What stays English).
  • Chinese translation covers the console’s own surfaces; help-topic bodies are translated, but the documentation pages they link to follow the docs site’s own language coverage.
  • Two server-side V3 metric bugs (nanosecond bucket-usage age, transposed bucket traffic) are tracked upstream and are not worked around in the console.
  • Automatic self-update remains disabled; upgrades are explicit.

Issues Closed

2.1.0 closes every issue filed against 2.0.0. Each carries a comment on the tracker describing the fix, the commits, and the coverage added.

Issue Resolution
#1 — unauthenticated deep routes recurse /login Absolute, base-path-aware login destination; deep-link and subpath test coverage
#2 — stale Uptime, malformed legends, cramped menus Uptime derived from real server state, legends resolve on the V3 name label, 32 px chart controls, popup width floors
#3 — 390 px viewport clips content Scrollable metrics tab strip; bucket table with a deliberate mobile column budget
#4 — unnamed collapsed sidebar buttons Labels visually hidden rather than removed from the accessibility tree; named, keyboard-operable collapse toggle
#5 — Access Key fields lack autocomplete metadata Field-level username / new-password tokens in a dedicated autofill section
#6 — English/Chinese localization Hand-rolled bilingual layer, zero new dependencies, English-as-key fallback
#7 — migrate monitoring queries to Metrics V3 V3-only; 26 widgets, 31 queries, 29 metric names, guard taxonomy, regression suite
#8 — replace N/A Info metrics Erasure Health and Usage Data Age, sharing the advanced dashboard’s widget results

Three acceptance criteria are recorded as unmet rather than quietly ticked: web-app has no unit-test runner, so the i18n test suite (#6) and the focused constructLabelNames test (#2) would require introducing test tooling first, and #6’s contributor documentation for adding translation keys is not yet written.

The complete v2.1.0 change set consists of 20 logical commits. The v2.1.0 tag additionally carries three later documentation commits that rewrote the repository README; they change no shipped behavior.

  • 8764f5d — fix(web): stop recursive login redirects
  • 437c56c — fix(ui): make the dashboard and bucket list usable on narrow screens
  • 85fc0c6 — fix(a11y): name collapsed sidebar controls and credential fields
  • e3fed07 — fix(metrics): rebuild dashboard cards, chart controls, and layout
  • fa11576 — feat(login): polish controls and legal attribution
  • 9fc17c1 — feat(i18n): add hand-rolled EN/ZH core, dictionaries, and language toggle
  • 622c02e — feat(i18n): localize login, navigation, and the help system
  • 6a03719 — feat(i18n): localize dashboard and metrics screens
  • 14b1c2d — feat(i18n): localize bucket and object browser screens
  • 0298062 — feat(i18n): localize identity, configuration, and event destinations
  • 41094f6 — feat(i18n): localize observability, admin tools, and shared components
  • e964992 — feat(metrics): migrate the dashboard to MinIO Metrics V3
  • 0b2251f — fix(i18n): harden the translation runtime for live data and chart legends
  • 9b60148 — fix(console): unify timestamps on a timezone-carrying standard format
  • bf110ae — fix(console): give selectable tables a visible-rows select-all
  • 5fc8f22 — fix(i18n): escape-proof all placeholder substitutions
  • fef8fab — fix(console): polish speedtest, sidebar, and help chrome
  • c4911e8 — chore(console): drop SUBNET remnants from health reporting
  • 1d631c4 — docs: record the SILO Console v2.1.0 changelog
  • 912d847 — build: regenerate optimized embedded web assets

Links:

2.9 - Silo Console 2.0.0 Released

The first independent major release of SILO Console: full identity and delivery migration, a redesigned login page and console UI, embedded assets cut from ~10MB to 3.5MB, zero known dependency vulnerabilities, and a batch of inherited bug fixes.

Published: 2026-08-04 · Version: v2.0.0 · Repository: pgsty/silo-console

SILO Console 2.0.0 is the first major release of this object-storage administration console as an independent project. Continuing from the georgmangold/console v1.9.1 maintenance line, it accomplishes three things:

  1. An independent identity — product name, visual system, documentation entry points, source attribution, and the release pipeline all move into the SILO project, while the Go module path, environment variables, and other compatibility contracts are deliberately retained;
  2. A redesigned interface — the login page, theme system, dashboard, and console details are reworked under one design language, backed by a regenerated brand icon set;
  3. Hardened engineering — the embedded frontend payload shrinks from roughly 10MB to 3.5MB, known dependency vulnerabilities drop to zero, and a batch of inherited defects — including a real runtime data race — is fixed.

Before publication this release went through two independent review passes: a full code review with commit-history restructuring, followed by an adversarial re-verification (exhaustive asset validation, HTTP semantics probing, full routing regression, and smoke tests against the published artifacts themselves).

Warning

Read the compatibility boundary before upgrading

The major-version change in 2.0.0 is about public identity and delivery contracts, not the object data format or the S3 protocol. Installation scripts that reference the old repository, binary name, or container image must be updated; existing integrations that use CONSOLE_MINIO_SERVER, CONSOLE_MINIO_REGION, github.com/minio/console, or the MinIO-compatible Admin API must not be search-and-replaced.

Why 2.0.0

This console originated as MinIO Console and was carried forward by the Alevsk/console and georgmangold/console community maintenance lines. SILO Console continues from there, maintained by the Pigsty community as the browser-based administration interface for SILO.

The version jumps from v1.9.1 to v2.0.0 because these public contracts change together:

  • the product is now uniformly SILO Console, with the primary repository at pgsty/silo-console;
  • the release binary changes from console to silo-console;
  • release assets, checksums, package metadata, CLI descriptions, and project links all switch to SILO;
  • in-product identity, help entry points, copyright attribution, source offers, and trademark notices are re-established.

The migration strategy is “clear external identity, restrained internal compatibility”: operators must take notice, but the underlying compatibility interfaces are not mechanically renamed.

Naming and Delivery Contracts

Scope Previous name or location 2.0.0 contract
Product Console / legacy MinIO Console SILO Console
Repository georgmangold/console pgsty/silo-console
Release binary console silo-console
Binary assets console-<os>-<arch> silo-console-<os>-<arch>
Checksums console_<version>_checksums.txt silo-console_<version>_checksums.txt
Website and docs upstream / previous maintainer silo.pgsty.com and silo.pgsty.com/docs/

For container deployments, use pgsty/silo-console on Docker Hub. See the Console v2.4.1 release notes for the current version and pull command.

CLI authorship, usage text, and project descriptions now identify Pigsty and SILO Console. DEB/RPM/APK vendor, maintainer, homepage, description, and license metadata are updated accordingly; the executable installs to /usr/local/bin/silo-console.

Deliberately Retained Compatibility Identifiers

The following names still contain minio or the old console, but they are interface, protocol, or installation compatibility layers — not leftover branding:

Surface State in 2.0.0 Reason
Go module github.com/minio/console retained changing it breaks every Go import
Server endpoint CONSOLE_MINIO_SERVER retained widely used by existing deployments
Server region CONSOLE_MINIO_REGION retained existing compatibility contract
Other configuration existing CONSOLE_* variables remain valid avoids migration with no benefit
S3/Admin API names MinIO-compatible fields and enums retained they describe the actual protocol
Development build make console still produces ./console keeps developer workflows working
Package systemd unit minio-console.service retained avoids duplicate services on upgrade
systemd user and config console-user and /etc/default/console avoids unnecessary account/config migration

Upgrade scripts therefore must not run repository-wide minio → silo or console → silo-console replacements. Migrating these compatibility interfaces in the future will require aliases, deprecation windows, and an explicit dual-read strategy; 2.0.0 does none of that.

A Redesigned Interface

2.0.0 is not a logo swap — the interface was redesigned end to end.

Login page

The login page is rewritten from scratch. The left brand panel renders a slowly drifting sine-mesh animation generated purely on Canvas (zero external dependencies, honors prefers-reduced-motion, pauses in background tabs), states the project’s proposition — “Keep the S3 Interface / Own the Object Store” — and keeps the full MinIO trademark notice at the bottom. The right-hand form is functionally untouched, preserving every existing automation selector. The Chakra Petch typeface used by the SILO wordmark ships as a ~20KB locally bundled subset with no external requests.

A unified theme system

All console colors converge into one light/dark theme layer: neutral greys for text and borders, the brand steel blue for primary actions and selection, and a sidebar that uses the same night palette as the login panel in both modes. Controls and cards share consistent radii and transitions, inputs get a keyboard focus ring, and modals animate in (also honoring reduced motion). Server-provided customStyles keep full precedence.

Console polish

  • Dashboard (Metrics): stat cards rebuilt under one grammar — muted labels, tabular numerals, aligned status dots; charts and info strips are theme-driven; the upstream absolute-positioning layout is gone.
  • Unified empty states: placeholder text in Watch, Trace, bucket Events/Replication/Lifecycle, and every other data panel is now centered and de-emphasized instead of raw top-left text.
  • Vertical tabs: detail-page tabs change from bordered grey blocks to a quiet pill list, eliminating the stray empty cell at the bottom of the rail.
  • License page: a new VERSION section shows both the connected server’s release and the Console’s own version; accounts without admin:ServerInfo never issue the request and the row stays hidden. The page also consolidates AGPLv3 licensing, the AGPL section-13 source offer, lineage, and trademark boundaries.
  • A batch of interaction fixes: the sidebar now collapses on initial load at mobile widths (previously it waited for a resize event); the bottom navigation no longer lags window-height changes; the bucket accordion highlight spans the full row; the dashboard no longer overflows horizontally on narrow screens; and the help panel is now truly lazy — the login page makes no external requests at all.

Brand icon set

The favicon, PWA, and Apple Touch icons still carried a previous-generation hand-drawn emblem. 2.0.0 re-rasterizes every size (ico 16+32, favicon 16/32/96, apple 180, manifest 192/512) from the official silo.svg vector emblem, with safe-area margins on home-screen sizes, and trims the Web App Manifest to the modern icon set, dropping the 2014-era legacy density entries. The icon payload drops from 473KB to 160KB, and the browser tab icon finally matches the in-product brand.

Smaller and Faster

Embedded delivery is this console’s core form factor — the frontend ships inside the binary via go:embed. 2.0.0 optimizes that path systematically:

  • Embedded payload: ~9.6MB → 3.5MB. Text assets (JS/CSS/SVG/…) are precompressed at build time with deterministic gzip and embedded compressed-only; legacy WOFF fonts (~1.25MB that no supported browser ever downloads) and a set of entirely unreferenced orphan images are removed.
  • First-load transfer: ~5.7MB → ~1.7MB. Static assets previously shipped uncompressed on the wire; they are now emitted directly with Content-Encoding: gzip at zero runtime cost, with on-the-fly decompression for the rare client that does not accept gzip.
  • Correct HTTP semantics. Accept-Encoding is parsed with full RFC 9110 q-values (gzip;q=0 gets identity bytes), responses carry Vary: Accept-Encoding, and non-GET/HEAD requests to static paths and the SPA entry receive 405 with an Allow header.
  • Reproducible builds. Compression uses a pure-JS implementation (fflate) for byte-identical output across platforms, and the release pipeline enforces a hard gate: rebuilding the embedded assets in a clean environment must produce zero diff against the commit.

Release binaries (all frontend assets included, stripped) weigh roughly 35–40MB; for the downstream SILO server, embedding this console now costs about 3.5MB instead of about 10MB.

Security and Dependencies

Go: the build baseline moves to Go 1.26.5 and the golang.org/x family is fully refreshed. Every reachable vulnerability reported by govulncheck is resolved:

Dependency Fixed version Advisories
google.golang.org/grpc v1.82.1 GO-2026-6061
github.com/prometheus/prometheus v0.311.3 GO-2026-5710 / -5662 / -5381 / -5264 (incl. remote-read DoS)
github.com/klauspost/compress v1.18.7 GO-2026-5841

The single remaining advisory sits in golang.org/x/crypto, has no upstream fix yet, and is unreachable from this codebase; it is tracked as a known item.

Frontend: the full dependency-tree audit (production and tooling) is clean, covering the high-severity form-data CRLF injection and the DOMPurify and qs advisories; React Router is migrated to 7.18.2 (keeping the v6-compatible declarative API, with full routing regression). The only explicitly ignored advisory affects an unstable API this project does not use.

Runtime correctness: a real data race between HTTP log-target initialization and shutdown is fixed, along with shared-mock races in the test suite; supported Go packages pass -race across the board. As a side benefit, the go-m1cpu upgrade fixes the local go run cgo crash on recent macOS.

Update Checks and Default Network Behavior

This release keeps conservative defaults for upgrade tooling:

  • automatic self-update in silo-console update is disabled — the command prints guidance and never downloads or replaces the binary;
  • the release catalog gains SILO_RELEASE_SERVICE_HOST, with the previous RELEASE_SERVICE_HOST as a compatibility fallback; with neither set, no remote release service is contacted;
  • the help panel’s blog content loads only when opened, and its links accept https://silo.pgsty.com exclusively.

Automatic updates will be reconsidered once signed release assets and a tested rollback path are in place.

Release Artifacts and Platform Matrix

The release ships 16 assets:

Type Coverage
Standalone binary Linux amd64/arm64/arm, macOS amd64/arm64, Windows amd64
System packages DEB / RPM / APK × amd64/arm64/armv6
Checksums silo-console_2.0.0_checksums.txt (SHA-256)

The pipeline triggers on tag pushes, pins third-party Actions to commit SHAs, and enforces the clean-checkout and zero-diff asset-rebuild gates before GoReleaser runs.

Upgrade Guide

Standalone binary

install -m 0755 silo-console-linux-amd64 /usr/local/bin/silo-console
/usr/local/bin/silo-console server

When building from source, make console still produces ./console; install it under the release name before wiring it into a production service.

DEB/RPM/APK and systemd

Packages continue to install /etc/systemd/system/minio-console.service, whose unit starts /usr/local/bin/silo-console. EnvironmentFile=/etc/default/console, console-user, and existing CONSOLE_* variables are unchanged. This retention lets package upgrades keep acting on the existing service instead of creating a parallel one.

Configuration and integrations

  • do not rename CONSOLE_MINIO_SERVER or CONSOLE_MINIO_REGION;
  • do not touch github.com/minio/console in Go imports;
  • prefer SILO_RELEASE_SERVICE_HOST for self-hosted release catalogs;
  • replace any reliance on console update with explicit download, verification, and deployment;
  • update process-path-based monitoring to /usr/local/bin/silo-console.

This release does not change the object data layout and requires no bucket or object migration.

Dual Review and Validation Scope

2.0.0 went through two independent review passes before publication. The first pass performed a full code review, fixed the defects described above, restructured 13 intermediate commits into 8 logical ones, and ran Go -race across supported packages, go vet, golangci-lint, govulncheck, frontend type checks, production builds, Prettier, dead-code checks, and the full dependency audit. The second, adversarial pass independently re-ran the core gates and added:

  • all 184 embedded files fetched three ways each (gzip client, identity client, HEAD) with per-file hash comparison against the embedded sources;
  • RFC semantics probes (including combined q-values such as gzip;q=0, *;q=0.5), method restrictions, the OIDC callback, and SPA deep links;
  • full React Router 7 regression: deep links, client-side navigation, bucket-detail tab switching, and browser history back;
  • mobile first-load sidebar behavior, login-page external-request monitoring, and light/dark full-site tours;
  • downloaded release assets verified byte-for-byte against checksums, binary self-reported version confirmed, and a smoke test of the published binary against a live server;
  • zero-diff asset rebuilds confirmed on both macOS and Linux.

The complete pre-rewrite history is preserved in backup refs for rollback.

Known Limitations

  • automatic self-update is disabled; upgrades are explicit;
  • the SSO end-to-end suite requires an external OpenLDAP/Dex/MinIO topology and was not run in that environment this cycle (the OIDC code paths are covered by unit tests and HTTP-level checks);
  • one golang.org/x/crypto advisory has no upstream fix yet and is unreachable from this codebase;
  • SILO does not yet maintain its own video library; videos in the help panel are clearly labeled upstream compatibility material;
  • administrative features depend on the MinIO-compatible Admin API — SILO Console is not a generic browser for arbitrary S3 services;
  • retained Go module paths, environment variables, protocol fields, and the systemd unit name still appear in code, configuration, and process listings.

The complete v2.0.0 change set consists of 8 logical commits:

  • 50797de — feat: establish SILO Console identity and compatibility
  • 23ae6e8 — feat: redesign and harden the SILO Console web app
  • 7a83a77 — build: update Go toolchain and dependencies
  • 1330d25 — fix: eliminate logger shutdown and test mock races
  • 06b3a34 — docs: publish the SILO Console v2.0.0 guide
  • 4b24372 — build: regenerate optimized embedded web assets
  • c38eb64 — ci: package and publish SILO Console v2 releases
  • b952a12 — brand: regenerate the icon set from the official silo.svg emblem

Links:

2.10 - silo-pkg 3.13.0 Released

silo-pkg 3.13.0 takes github.com/pgsty/silo-pkg/v3 as its own module path, retires the replace-directive arrangement it depended on, and drops the Silo Go SDK fork in favour of upstream minio-go.

Follow-up, 2026-09-13: this article preserves the v3.13.0 release-time migration record. All four maintained components have since adopted the own-module path; current pkg is v3.14.0. The “not yet migrated” statements below describe that earlier date; see the current component matrix.

Release date: 2026-08-30 · Version: v3.13.0 · Commit: 215f116 · Repository: pgsty/silo-pkg

Version 3.13.0 is a breaking release with one theme: the module stops claiming upstream’s import path and takes its own. It also retires the pgsty/silo-go fork, which no longer carried anything upstream lacks.

Warning

This package release and a SILO server release are different gates. The package tag and GitHub Release are public. silo, silo-console, and mcli have not yet moved to the new import path; they continue to build against v3.12.2 through the old replace arrangement. No SILO server tag, container image, package set, deployment, or production rollout is established by this article.

Release at a Glance

  • Published: the silo-pkg v3.13.0 tag and GitHub Release, declaring module github.com/pgsty/silo-pkg/v3.
  • Retired: the github.com/minio/pkg/v3 module identity and the replace directive every consumer had to repeat.
  • Retired: the github.com/pgsty/silo-go/v7 fork; this release requires upstream minio-go directly.
  • Unchanged: every package, symbol, and behaviour. Only the path they are imported from moved.
  • Not part of this release: the matching silo, silo-console, and mcli migrations.

Why the Path Moved

The fork kept upstream’s github.com/minio/pkg/v3 path so it stayed a drop-in replacement, selectable with one replace directive. Go does not inherit replace directives from dependency modules, and three costs came due while preparing the next SILO server release.

Every consumer had to repeat the redirect, and one that forgot built against upstream silently. A module that requires this package but omits the replacement resolves the real minio/pkg, compiles, and quietly loses the fork’s behaviour.

The require line had to name a version the source no longer matched. ParseConfigStrict first appears upstream in v3.11.0, and Resource.IsBareARN exists in no upstream version at all. A require github.com/minio/pkg/v3 v3.6.1 next to source that needs v3.12 APIs is metadata that cannot be made true by editing the number.

Working around that propagated. pgsty/mc carried a compile-time sentinel — a reference to Resource.IsBareARN — purely to turn the silent downgrade into a build failure. Raising mc’s floor to make its metadata honest pushed the requirement through the module graph and forced silo-console’s deliberately low floor upward, which is exactly what that floor existed to prevent.

Upstream’s path had little to offer in return here. minio/pkg is a small internal library, and this module’s consumers are silo, silo-console, and mcli. The Silo Go SDK keeps upstream’s github.com/minio/minio-go/v7 path, where drop-in compatibility is worth having and upstream is actively maintained.

Upstream minio-go Replaces the Silo Go SDK

pgsty/silo-go no longer carried any functional divergence. Its one unique change, Return CopyObject checksums in UploadInfo, was merged upstream on 2026-08-24. Everything else in the fork was a version string, a logo, a README, and a lint-tool block.

Upstream’s newest tag v7.3.0 predates that merge by 14 commits, among them a data race fix in parallel multipart checksum hashing, so pinning the tag would be a regression. This release therefore requires the pseudo-version v7.3.1-0.20260828014306-0e78d3f18efe and will move to a tag when upstream cuts one.

Compatibility

Every package, exported symbol, and behaviour is unchanged. This release moves where they are imported from, nothing else.

Take it together with the matching silo, silo-console, and mcli changes. A consumer that upgrades alone will not build, because the old and new paths are different modules and types do not cross between them.

Before:

require github.com/minio/pkg/v3 v3.6.1

replace (
    github.com/minio/pkg/v3      => github.com/pgsty/silo-pkg/v3 v3.12.2
    github.com/minio/minio-go/v7 => github.com/pgsty/silo-go/v7 v7.3.1
)

After:

require github.com/pgsty/silo-pkg/v3 v3.13.0

…and rewrite github.com/minio/pkg/v3/... imports to github.com/pgsty/silo-pkg/v3/.... Consumers that stay on the old arrangement keep building against v3.12.2 and earlier, which remain published.

One consequence is worth expecting. Because the two paths are now separate modules, a build can contain both — a third-party dependency that imports github.com/minio/pkg/v3 no longer has its import redirected. In the SILO stack this happens once, through minio/colorjson and minio/dperf, both of which reach only pkg/v3/console. That package’s colour switch is fatih/color’s process-wide NoColor, which every copy shares, so disabling colour still disables it everywhere.

Verification

  • go build ./..., go vet ./..., go mod tidy -diff, and gofmt -l . are clean; go test ./... passes 23 packages.
  • CI on the release commit: Go, LDAP Config Validator, VulnCheck.
  • The new path was resolved from the module proxy by a fresh module with no replace directive, which imported policy and ran.
  • The downstream migration was validated end to end before this release: mcli (194 files), silo-console (36 files), and silo (181 files) all build, vet, and pass their suites. silo’s rebrand guard passes with a compatibility-baseline diff of exactly 19 deleted import entries and no change to environment variables, metrics, headers, routes, policy values, or exported symbols.

2.11 - silo-pkg 3.12.0 Released

silo-pkg 3.12.0 rejects bare ARN namespace prefixes on strict policy-write paths, preserves legacy policy loading, moves the maintained baseline to Go 1.27 and etcd 3.7.1, and ships the reviewed SILO dependency stack.

Release date: 2026-08-24 · Version: v3.12.0 · Commit: 2b087a1 · Repository: pgsty/silo-pkg

Version 3.12.0 is a main-line minor release with two themes: a policy-write guard for ARN prefixes that name no resource, and the maintained Go 1.27 / etcd 3.7 dependency baseline. It adds two exported inspection/validation methods, raises the verified consumer floor to Go 1.26, and is the first silo-pkg release whose strict validation path is enabled by the SILO server for named-policy and service-account policy writes.

Warning

This package release and a SILO server release are different gates. The package tag and GitHub Release are public. SILO main consumes it in eee05a17c, with the operator note recorded as SN-2026-005. No date-style SILO server tag, container image, package set, deployment, or production rollout is established by this article.

Release at a Glance

  • Published: the silo-pkg v3.12.0 tag, GitHub Release, and strict library validation for bare ARN prefixes.
  • On SILO remote main: named-policy and service-account policy write integration.
  • Deliberately permissive: existing IAM policy loading, IAM import, and site-replication receive paths.
  • Deferred: STS inline-policy strict validation and Console-side early validation; the server remains authoritative.
  • Not part of this release: a SILO server binary, image, package, deployment, or production rollout.

Bare ARN Prefixes Were Policy No-ops

An S3 resource ARN needs a resource after its namespace prefix:

arn:aws:s3:::my-bucket
arn:aws:s3:::my-bucket/*
arn:aws:s3:::*

The policy parser also accepted the prefix by itself:

arn:aws:s3:::

That string names no bucket or object. The existing parser normalized it to the wildcard resource type while retaining arn:aws:s3::: as the match pattern. Real S3 authorization candidates look like bucket or bucket/object, so the pattern normally matched nothing even though the policy validated successfully.

For statements that actually perform resource matching, the impact depends on Effect and on whether the prefix appears in Resource or NotResource:

Statement shape Existing runtime result
Allow + bare Resource Grants nothing
Deny + bare Resource The intended denial does not fire
Allow + bare NotResource Excludes nothing and can grant far more than intended
Deny + bare NotResource Can deny far more than intended

The dangerous cases are policy-dependent fail-open behavior, not an unauthenticated remote exploit and not a CVE. A policy author, template, or automation must first submit the malformed resource. The same issue applies to registered S3 Tables and KMS ARN prefixes.

Resource-less admin actions, sts:* action statements, and the first phase of two-step KMS authorization bypass resource matching; a bare prefix does not change their existing runtime decision. Strict writes still reject the deceptive field so a policy cannot look scoped when that scope is ignored.

The Historical *arn:... Spelling

On the permissive compatibility path inherited from earlier releases, serializing a parsed bare prefix adds the wildcard type marker:

arn:aws:s3:::  ->  *arn:aws:s3:::

Re-parsing either spelling produces the same internal resource value. The 3.12 guard therefore recognizes both the exact prefix and its historical star-prefixed serialization. This matters for stored/exported policies and for clients that parse and marshal a document before sending it to the server.

The accepted wildcard corpus remains unchanged: *, **, ***, *foo, and explicit resources such as arn:aws:s3:::* still parse as before.

A Strict Write Path, Not a Storage Migration

The fix deliberately separates policy loading from policy creation:

  • ParseConfig and Validate remain permissive. Existing stored policies keep loading and evaluating with the same matching and serialization behavior.
  • ParseConfigStrict and ValidateStrict reject a registered ARN prefix that names no resource, in both Resource and NotResource.
  • Resource.IsBareARN() detects the normalized exact/historical form without changing the exported Resource structure, ParseResource, matching, or JSON representation.
  • ResourceSet.ValidateStrict() exposes the strict resource-set check to consumers.

Keeping the existing resource representation is important for mixed-version sites: v3.11 and v3.12 nodes continue to compare and serialize stored policies the same way, so this fix does not create a site-replication mismatch or require a storage migration.

SILO Enables the Guard on Three Boundaries

SILO commit eee05a17c selects silo-pkg v3.12.0 and uses strict parsing when:

  1. creating or replacing a named IAM policy;
  2. creating a service account with an inline session policy; and
  3. updating a service account’s inline session policy.

Compatibility-sensitive paths stay permissive in this rollout:

  • loading named policies and embedded policies already at rest;
  • IAM import/restore;
  • site-replication receive and apply paths;
  • STS inline session policies; and
  • bucket policies, whose existing bucket/action validation already rejects these forms.

Enabling ParseConfigStrict also activates two pre-existing admin-policy checks: one admin statement may not contain both Resource and NotResource, and a bucket-scoped admin action may not use a non-S3 resource. These are intentional authorization tightenings and are documented in the advisory ledger under SN-2026-005.

Important

In this article, bare ARN prefix means an ARN namespace with no resource after it, such as arn:aws:s3:::. It is different from the valid bare bucket ARN used in the 3.11 bucket/object-boundary fix, such as arn:aws:s3:::my-bucket.

Operator Action

Existing policies are not rewritten automatically because the intended resource cannot be inferred. Before deploying a SILO server build that contains the strict integration:

  1. inspect named IAM policies for exact or historical bare prefixes;
  2. inspect service-account inline policies;
  3. replace each finding with the intended concrete resource; or use a suffix wildcard only if all resources in that namespace are truly intended; and
  4. repeat the audit after all sites have completed the rolling upgrade.

Do not blindly turn every finding into arn:aws:s3:::*: that could replace an inert statement with a cluster-wide grant or denial. A legacy policy containing a bare prefix still loads, but it cannot be submitted unchanged through the three strict write endpoints; correct it before editing another property on the same policy or service account.

The safer audit path uses policy-info and access-key-info APIs rather than a full IAM export, because a complete export contains user and service-account secrets. STS strict validation remains deferred until live machine clients and their session-policy templates can be audited separately.

Why This Is a Minor Release

This is v3.12.0, not v3.11.1, because the release combines three compatibility-relevant changes:

  1. the etcd client crosses from the 3.6 minor line to 3.7;
  2. the module’s verified go floor rises from 1.25.0 to 1.26.0; and
  3. the package adds exported bare-ARN inspection and strict resource-set validation APIs.

The module path remains github.com/minio/pkg/v3; the /v3 import suffix and every existing import site stay unchanged.

Go and Dependency Baseline

The go and toolchain directives have separate purposes:

  • go 1.26.0 is the supported consumer floor required by the selected etcd 3.7 modules.
  • toolchain go1.27.0 is the maintained development and CI baseline.
  • CI actions move to the Node 24 runtime.

Key selected versions change as follows:

Module 3.11.0 3.12.0
go.etcd.io/etcd/{api,client/pkg,client}/v3 3.6.6 3.7.1
golang.org/x/crypto 0.54.0 0.55.0
golang.org/x/net 0.57.0 0.58.0
golang.org/x/text 0.40.0 0.41.0
github.com/minio/minio-go/v7 7.0.97 7.0.99
github.com/minio/mux 1.8.2 1.9.2
github.com/cheggaaa/pb 1.0.29 1.0.30
github.com/lestrrat-go/jwx/v3 3.0.12 3.0.13
github.com/lestrrat-go/httprc/v3 3.0.1 3.0.6
github.com/grpc-ecosystem/grpc-gateway/v2 2.27.3 2.29.0
go.uber.org/zap 1.27.1 1.28.0

Smaller selected updates include uax29 2.3.1, fastjson 1.6.10, secp256k1 4.4.1, and goccy/go-json 0.10.6. The old lestrrat-go/option v1, gogo/protobuf, and stale test-only requirements leave the selected graph.

etcd 3.7.1 is the first fixed release on the 3.7 line for GO-2026-6107 / CVE-2026-73500, an unauthenticated TLS-listener denial of service. Updating these Go client modules does not upgrade an operator’s external etcd server. SILO does not use the removed grpc.WithBlock behavior, and client compatibility with an etcd 3.6.14 server was exercised during release validation.

The dependency graph declares coreos/go-systemd 22.7.0, but the module replaces it with 22.6.0 because 22.7.0 does not compile on NetBSD. The SILO server carries the same portability override.

Compatibility

  • Import paths and the module major remain unchanged.
  • Existing policies keep loading and evaluating unchanged; only strict create/update calls reject the malformed prefixes.
  • No policy, IAM database, wire protocol, or etcd data migration is performed.
  • Downstream users of grpc.WithBlock in etcd client dial options must migrate to a supported readiness check; SILO does not use it.
  • A real external etcd cluster upgrade remains a separate operational procedure.
  • Upstream AIStor Memory and new AIStor-only S3 Tables action vocabularies are not imported by this fork.

Consumers select the release with:

require github.com/minio/pkg/v3 v3.12.0

replace github.com/minio/pkg/v3 => github.com/pgsty/silo-pkg/v3 v3.12.0

Verification

The tagged package passed:

  • the repository’s complete make test gate: pinned lint plus go test -race -tags kqueue ./...;
  • targeted bare-ARN tests repeated to disturb Go map iteration order;
  • go mod verify, go vet, git diff --check, and govulncheck with zero reachable vulnerabilities; and
  • an implementation-level Claude Opus Max review with a GO verdict and no P0/P1 findings.

The SILO integration passed:

  • the complete IAM server suite, including exact/historical named-policy rejection and service-account create/update rejection;
  • go test ./cmd -count=1, go vet ./cmd, and go test ./...;
  • golangci-lint 2.13.1 with zero findings, go mod verify, and make check-gen; and
  • a second Claude Opus Max review with a GO verdict; mutation tests proved all three strict call sites and their integration assertions are load-bearing.

Direct VCS module resolution verified v3.12.0 at commit 2b087a1 with module checksum:

h1:1Bjqjb3KCt0oYhBLpH7W/e/5khTUoIgXWA12An1fbUc=

The release environment could not reach proxy.golang.org or sum.golang.org because those connections timed out, so public-proxy observation is not claimed as release evidence. The Git tag, GitHub Release, direct module archive, origin commit, and checksum were verified.

  • 2bc3a91: move CI actions onto Node 24
  • c8c6872: align the SILO Go dependency stack
  • 2b087a1: reject bare ARN prefixes on strict policy writes; tagged v3.12.0
  • 30c49bd: update the README dependency example after the tag
  • eee05a17c: enable strict named-policy and service-account writes in SILO
  • 56c67dacf: record SN-2026-005

What Is Not Released Here

This article does not claim a new SILO server version, binary, package, container image, deployment, production rollout, Console release, or strict STS rollout. Those remain separate release gates and must be reported separately when completed.

2.12 - Silo Pkg 3.11.0 Released

The fork’s first pinned release restores the IAM bucket/object resource boundary that let an object-only grant reach bucket-level writes, fixes a policy condition-key bypass and three LDAP connection defects, and renumbers onto upstream’s 3.11 line after the earlier tags were found to collide with upstream releases of the same numbers.

Release date: 2026-08-04 · Version: v3.11.0 · Commit: d8b1fa7 · Repository: pgsty/silo-pkg

This is the fork’s first pinned release. It restores the IAM bucket/object resource boundary reported as upstream minio/minio#20449: a policy condition-key bypass fix, three LDAP connection defects, a certificate watcher leak, a seeded-RNG defect, and the module’s real minimum Go version.

Warning

Two things to check before upgrading

  1. This release tightens authorization. Twelve bucket-level write actions are no longer reachable through an object-only resource pattern such as arn:aws:s3:::bucket/*. If you write your own bucket-scoped policies, read The IAM bucket/object boundary — the fix is one line of policy for anyone affected, and MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on restores the previous behaviour in full.
  2. The condition-key fix still needs its server half. The policy lookup change and the server changes that reserve internal condition-key names each cover one half of that problem. The companion server work exists in pgsty/minio commit 2f55347f7 but is not yet on public origin/master, and no published Silo server release contains it. Verify that a later server release explicitly includes it.

What This Repository Is

silo-pkg is a maintained fork of minio/pkg, carrying fixes needed by community MinIO forks that the now commercially driven upstream no longer accepts. The repository was renamed from pgsty/minio-pkg on 2026-08-02.

The module path intentionally remains unchanged as github.com/minio/pkg/v3. Existing import "github.com/minio/pkg/v3/..." statements do not change; only the right-hand side of the replace directive does:

replace github.com/minio/pkg/v3 => github.com/pgsty/silo-pkg/v3 v3.11.0

The /v3 suffix is the module’s major version, not a directory name, and must not be omitted. It is also why this release is numbered v3.11.0 rather than v4.0.0: Go requires the major version of a tag to match the major-version suffix declared in go.mod, so a v4.0.0 tag on a .../v3 module is rejected by the toolchain. Publishing a real v4 would mean changing the module path and rewriting roughly 395 import sites across the server, mc and Console — abandoning the drop-in property that is the point of keeping upstream’s path.

The IAM Bucket/Object Boundary

Every bucket-level S3 operation authorizes with an empty object name. The IAM matcher turned that into a resource string and, for the empty-object case, appended a trailing slash:

resource.WriteString(args.BucketName)
if args.ObjectName != "" {
    // "bucket/object"
} else {
    resource.WriteByte('/') // "bucket/"  <-- the defect
}

"bucket/" is matched by the wildcard pattern "bucket/*", because * matches the empty string. A policy granting s3:* on arn:aws:s3:::bucket/* — which reads as “anything, but only on the objects in this bucket” — therefore also authorized bucket-level actions. In a multi-tenant cluster, a tenant holding only that grant could call PutBucketPolicy and install {"Principal":"*"}, making the bucket publicly readable or writable, or grant itself bucket-level control. It could also delete the bucket outright, which is the reproduction in the upstream issue.

The bucket-policy evaluation path used for anonymous access never had this slash and was already reference-correct. Only the IAM path was wrong, in exactly one place.

Why not correct the whole boundary

Removing the slash for every bucket-level request is the obvious fix, and upstream tried it: the change was reverted the same day for breaking policies that relied on the old behaviour. Two properties make the full correction a migration rather than a patch.

It revokes grants real deployments depend on. It does not only revoke the dangerous bucket writes — it also revokes ListBucket, GetBucketLocation and ListBucketMultipartUploads when granted through bucket/*. The evidence is upstream’s own test suite: eleven STS integration tests grant s3:ListBucket on bucket/* and then assert that listing works. If the project that wrote the server writes it that way, production policies do too.

It cuts both directions. The matcher builds the same resource string for Allow and Deny, so removing the slash tightens over-granting Allow statements and simultaneously loosens over-blocking Deny statements. An administrator who locked a bucket with Deny s3:* on bucket/* would silently lose that protection.

How the protected set was chosen

The scope was decided by one question: does reaching this action give the caller something its object-scoped grant does not already provide?

That question is the right one because of how the defect fires. Resource matching runs after action matching, so the bug only bites when the statement already grants the bucket-level action — which in practice means s3:*. The affected principal therefore already holds full read, write and delete over every object in the bucket. The useful question is not how dangerous an action sounds in the abstract, but what reaching it adds to a position that already includes all of the data.

Withheld from object-only grants (twelve actions):

Action Why it qualifies
PutBucketPolicy, DeleteBucketPolicy Hand access to other principals, anonymous included, and can grant the caller bucket-level actions it was never given. Self-escalation and public exposure.
PutBucketObjectLockConfiguration, PutBucketVersioning Defeat protections that exist precisely to stop a holder of write access from destroying data.
PutReplicationConfiguration, PutLifecycleConfiguration Act under server credentials and keep acting after the caller’s access is revoked.
DeleteBucket, ForceDeleteBucket Destroy the bucket entity and its configuration irreversibly. The reproduction in the upstream issue.
PutBucketCors, DeleteBucketCors, PutBucketQOS, PutInventoryConfiguration No server behaviour is attached to these today — no handler at all, or a handler that returns NotImplemented after the authorization check. Withholding them costs nothing and covers them in advance.

Deliberately not withheld, and asserted by a test so that adding one is a deliberate act with a visible cost rather than an edit to a list:

  • PutBucketTagging, PutBucketEncryption, PutBucketNotification. These are bucket-level writes and an earlier draft did withhold them. None gives the caller access it does not already hold — the harm is to the owner’s posture, not to the access boundary — while a tenant handed s3:* on bucket/* and told the bucket is theirs may quite reasonably tag it, set default encryption, or wire up event notifications. Low security gain against a real compatibility cost is the wrong trade for a maintenance release.
  • CreateBucket. It targets a bucket that does not exist yet, so there is nothing to mutate or destroy, and provisioning flows commonly create a tenant’s bucket with that tenant’s own bucket/* credentials.
  • The read/list family (ListBucket, GetBucketLocation, the configuration reads). Breaking these is what got upstream’s own attempt reverted. They wait for a migration-gated release.

Only Allow statements are affected. Deny statements keep the historical resource string, so no bucket lock is ever weakened, and NotResource exclusions keep their full reach.

Monotonicity, and the claim that was wrong twice

All of the above rests on one property: this change may remove permissions and must never add one. That property was asserted twice from reasoning rather than from tests, and was false both times. Recording how is more useful than recording only the final state.

The first attempt let the withheld slash reach the NotResource match as well — and NotResource is an exclusion. An Allow s3:* NotResource bucket/* statement historically did not apply to bucket-level requests on that bucket; matching the exclusion against the bare bucket name made it stop matching, so the Allow it qualified grew, for exactly the writes being protected.

The second attempt fixed that and shipped saying the result was provably monotone. An independent adversarial review of that release produced a counterexample. Withholding the slash does not merely remove a match — it changes which string patterns are matched against, and a pattern can match "mybucket" without ever having matched "mybucket/". A fixed-width wildcard is the clean case:

Allow s3:PutBucketPolicy on arn:aws:s3:::mybucke?

? matches exactly one character. Against the nine-character "mybucket/" it does not match, so this statement never authorized the bucket-level write. Against the new eight-character "mybucket" it does, so the hardening granted something the buggy matcher refused.

The fix is not another special case. On the protected path the matcher now requires both forms to match — the bare bucket name and the historical "bucket/". The result is an intersection with the historical decision, so it is monotone by construction: there is no pattern it can newly satisfy, and no argument left to get wrong. mybucket* still grants (it matched both all along), mybucket/* is still withheld, and mybucke? is refused exactly as it always was.

Two lessons are worth carrying forward. A correctness fix in an authorization path must never make anything newly allowed — and the only way to know is to test both directions, because the reasoning felt airtight in both cases where it wasn’t. And when a security property is load-bearing, build it out of an operation that cannot violate it rather than out of a case analysis believed to be complete.

Evidence

The property is verified rather than argued. A decision corpus of 27,000 authorization outcomes — 15 resource patterns × 3 buckets × 5 object names × 20 actions × 6 statement forms — was generated against both the pre-hardening baseline and this release and compared entry by entry:

Transition Count
false → true (broadening) 0
true → false (narrowing) 144
unchanged 26,856

Every one of the 144 narrowed outcomes falls inside the design intent, with nothing outside it: exactly the twelve protected actions; only the three Allow statement forms, with zero transitions for Deny, NotResource-excluded or deny-NotResource forms; only four object-only resource patterns; and only bucket-level requests, with object-level requests entirely untouched. 12 × 4 × 3 = 144, fully accounted for.

Regression coverage exists at both layers. In this repository, twelve matcher tests pin each direction, including an invariant test that every protected action really is bucket-only — ResetBucketReplicationState, despite its name, is an object action and stays out. In the server, three end-to-end tests drive the real handlers at the client, inline-session-policy and S3-router levels; all three fail against the pre-fix build and pass against this one.

What to change

You are affected only if a stored policy grants one of the twelve actions — or s3:* — on a resource pattern containing /, with no bare bucket ARN for the same bucket. The fix is to add the bare ARN alongside the object pattern:

"Resource": ["arn:aws:s3:::bucket", "arn:aws:s3:::bucket/*"]

That pairing is the conventional form, is what upstream’s own tests use, and worked before this release as well. Built-in canned policies are unaffected — readwrite, readonly, writeonly and diagnostics all use Resource: "*".

MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on, read once at startup, restores the historical matching in full — both the over-granting and the over-blocking. It is a single global switch; per-action scoping is deferred.

Policy Condition-Key Lookup Order

getValuesByKey() previously looked up a policy condition key by its canonical MIME spelling (http.CanonicalHeaderKey) before trying the original name. The map it reads mixes values calculated by the server for the current request (SourceIp, SecureTransport, CurrentTime, username and others, stored under condition-key spellings) with HTTP headers supplied by the request (stored under canonical MIME spellings).

Checking the canonical spelling first allowed a client header to override a value calculated by the server.

For a MinIO server this is a policy bypass. The simplest example is s3:prefix: a Prefix request header could satisfy a home-directory prefix condition while the real ?prefix= query parameter still listed the entire bucket. The same path reached aws:SourceIp, aws:SecureTransport, aws:CurrentTime, aws:EpochTime, aws:username, aws:userid, aws:principaltype, aws:UserAgent, aws:groups, ldap:username, ldap:groups, jwt:groups, s3:versionid, s3:signatureversion, s3:signatureAge, s3:authType and s3:LocationConstraint. Anonymous bucket policies were directly exposed. SigV4 did not prevent the attack because a client can add a header that is not listed in SignedHeaders.

There was a second consequence: when the server stored a value under one spelling and the policy key resolved another, the wrong entry won. s3:object-lock-mode could resolve to the caller’s X-Amz-Object-Lock-Mode header rather than the retention mode the server would actually apply.

The fix reverses the lookup order: match the condition key’s exact name first, then use the canonical spelling only as a fallback for condition keys that genuinely name request headers, such as the s3:x-amz-* family. This ports minio/pkg#226 and adds regression coverage the upstream change did not carry.

At the library’s raw-map layer, if a producer stores one logical field under both the exact condition name and its canonical MIME name, the exact name now wins. This is a library lookup rule, not an S3 wire-protocol rule that says query parameters take precedence. The Silo server first normalizes condition values by their real source. For storage class and upload tagging, where both Header and query forms remain compatible, Header presence wins, including an empty value; query is only the fallback.

LDAP Connection Path

Three defects in connect(). Two were introduced by this fork in b0c08a7 and shipped in v3.6.2 and v3.6.3. Users of either release should upgrade promptly.

StartTLS was skipped when ServerInsecure was enabled. Upstream called StartTLS in an outer block controlled only by ServerStartTLS, so enabling both options created a plaintext connection and then upgraded it. b0c08a7 moved the call into an else branch, making StartTLS unreachable whenever ServerInsecure was true. The connection stayed plaintext and the following bind sent credentials over it. MinIO exposes MINIO_IDENTITY_LDAP_SERVER_INSECURE and MINIO_IDENTITY_LDAP_SERVER_STARTTLS independently and Validate() rejects no combination, so this state was reachable.

This release restores the upstream semantics: the two switches are additive, not mutually exclusive. ServerInsecure disables implicit ldaps://; ServerStartTLS still performs the upgrade. The exposure window is limited to v3.6.2 and v3.6.3.

A Config without a TLS section could panic on the ldaps:// path. After l.TLS.Clone() moved outside the StartTLS branch, ordinary ldaps:// connections also called it. Clone() returns nil for a nil receiver, but the next line assigned ServerName. The MinIO server always supplies TLS settings, but this is a library and mc also consumes it. The code now falls back to an empty tls.Config, matching what DialURL would have built.

StartTLS had no deadline. go-ldap only starts its request timer when requestTimeout > 0, while StartTLS itself has no timeout. A server that completed TCP setup and then stopped responding to the extension request could hold the connect goroutine forever. The timer is now armed before StartTLS.

A failed StartTLS leaked the connection. Inherited from upstream. Dial failures do not return a connection, making StartTLS failure the only connect() path that could return both a connection and an error. Callers only took ownership when the error was nil, leaving a socket behind for every login attempt against a server with a broken upgrade. The failure path now closes the connection and returns nil.

Other Fixes

  • certs: file watchers were never stopped. Manager.AddCertificate() registered two notify.Watch() calls and stopped neither: if the second failed, the first leaked, and both survived until process exit after the manager closed. Certificate.Watch() and watchFile() had the same problem. All four paths now use watchDirSafe(), which returns a stop function invoked on errors and ctx.Done(). This ports the certs/ part of minio/pkg#228. On Windows the function replaces filesystem notification with polling rather than using polling only as a failure fallback, so certificate reload can lag by one symlinkReloadInterval (10 seconds). This fork has no Windows CI; that platform was only cross-compiled.
  • rng: reader subkeys came from a zeroed local variable. init() read 32 bytes of entropy into r.tmp but derived four subkeys from a same-named zeroed local, collapsing four per-block streams into one. Reset() and ResetSize() then replayed the previous stream byte for byte. MinIO creates a new reader for each randreader.New() call and never resets it, so the practical server impact is limited; warp exposed the defect. This ports minio/pkg#230.
  • xtime: Duration implemented UnmarshalJSON but not MarshalJSON. Encoding produced an integer number of nanoseconds while decoding unconditionally stripped the first and last byte and expected a quoted string, so neither direction could round-trip. It now encodes using time.Duration’s string form. This ports minio/pkg#242.

Compatibility Impact

  • Twelve bucket-level write actions are no longer authorized through an object-only resource pattern. See What to change. Object access, ListBucket, CreateBucket, bucket tagging, default encryption and event notification are all unaffected, as are Deny statements and NotResource exclusions.
  • The minimum Go version moves from 1.26.1 down to 1.25.0. A patch number in the go directive is a hard minimum for every consumer, not a record of the toolchain used to build the module. The conventional split is a language version on the go line and a development version on a separate toolchain line. 1.25.0 is what the dependency graph actually requires and what upstream declares. CI builds the complete test suite with Go 1.25 under GOTOOLCHAIN=local, so the minimum is proven rather than aspirational.
  • The JSON wire format of xtime.Duration changes from a nanosecond integer to a duration string such as "2h" or "30m". Persisted numeric values can no longer be read back. No such use was found in MinIO or mc: batch job definitions persist as YAML and the msgp path remains int64.
  • Deployments with both ServerInsecure and ServerStartTLS enabled whose LDAP server does not support StartTLS connected successfully in plaintext on v3.6.2/v3.6.3 and now fail to connect. That is the correct result, but it surfaces during connection rather than configuration validation. Disable ServerStartTLS for such a server.
  • Policy.IsAllowedActions can disagree with a direct decision for the twelve protected actions. It enumerates SupportedActions, which includes the s3:* pattern itself, so the returned set can contain s3:* — and therefore appear to permit a protected action — while the direct evaluation denies it. Nothing in the server calls it, and Console calls it with an empty bucket name, which never reaches the hardened branch. Recorded rather than changed, because altering a public API’s output in a maintenance release is the larger risk.

Divergence from Upstream v3.11.0

The version number follows upstream’s line and makes no claim of identical content. The measured delta, comparing action-string constants across policy/:

Count
Upstream minio/pkg v3.11.0 291
silo-pkg v3.11.0 270

24 actions exist only upstream: six s3:*ObjectAnnotation* actions, five admin: actions (DistJobStatus, Get/SetBucketCompression, two TablesReplication*), and thirteen s3tables: actions covering function CRUD and tagging. These belong to the AIStor vocabulary this fork deliberately does not carry, because the community server does not implement them.

Three actions are named differently on each side. Upstream renamed and split these; this fork retains the earlier names:

silo-pkg v3.11.0 upstream minio/pkg v3.11.0
s3tables:TagResource s3tables:TagTable, s3tables:TagWarehouse
s3tables:UntagResource s3tables:UntagTable, s3tables:UntagWarehouse
s3tables:ListTagsForResource s3tables:ListTagsForTable, s3tables:ListTagsForWarehouse

A policy naming any of these six action strings therefore validates on exactly one of the two. Nothing in the Silo server, mc or Console references them, so there is no impact inside this ecosystem — but a consumer swapping upstream v3.11.0 for this release should know the vocabulary is not interchangeable.

rng has no arm64 assembly. Upstream added rng/xor_arm64.{go,s} after this fork’s divergence point; this release falls back to the pure-Go xor_noasm.go path on arm64. The result is correct and cross-compiles cleanly, but slower than upstream on that architecture. It is a clean candidate for a future sync, being a pure performance change with no vocabulary entanglement.

Companion Server Behavior

  • The condition-key change in this release must be paired with the server changes that reserve internal condition-key names and populate values by semantic source, as noted at the top.
  • s3:signatureAge is exposed only after the SigV4 presigned-request verifier calculates it. A client-supplied x-amz-signature-age Header is ignored on every other request type.
  • s3:prefix, s3:delimiter and s3:max-keys come only from query parameters. Content hash, copy source, metadata directive, SSE and object-lock conditions come only from the corresponding headers. The X-Amz-Content-Sha256 query value consumed while verifying a presigned request does not become a policy condition.
  • s3:x-amz-storage-class retains its compatible query form, as do request tags on PutObject and CreateMultipartUpload. For both fields, Header presence wins and query is used only when the Header is absent.
  • s3:ExistingObjectTag/* comes only from tags loaded from the stored object, so a request’s own X-Amz-Tagging can no longer impersonate existing object state. PutObject, CreateMultipartUpload and PutObjectTagging bind s3:RequestObjectTag/* to the tag input those handlers consume. Other action paths retain the historical X-Amz-Tagging Header fallback for compatibility, so treat request-tag conditions as constraints only where the API actually consumes tags.
  • aws:SourceIp is calculated from forwarding headers. Whether it is enforceable depends on the server’s trusted-proxy configuration; see the server’s own release notes for MINIO_API_TRUSTED_PROXIES.

Verification

Everything below was run against the tagged commit, with the working tree clean and the tag pointing at HEAD:

  • make test — golangci-lint plus go test -race -tags kqueue ./..., all packages passing.
  • go mod tidy -diff clean; gofmt -l empty; go vet ./... clean.
  • Cross-compilation for linux/amd64, linux/arm64, darwin/arm64 and windows/amd64.
  • govulncheck ./... — zero reachable vulnerabilities. One module-level notice remains, GO-2026-5932 in x/crypto/openpgp; that package is unmaintained, has no fixed version, and this repository does not import it.
  • Resolution from an empty module cache through the public proxy, confirming the release is fetchable as published.
  • The 27,000-outcome authorization corpus described above.

Dependencies and Tooling

Dependency updates clear nine reachable findings previously reported by govulncheck: seven x/crypto/ssh issues reached through sftp, GO-2026-6061 in gRPC reached through etcd, and GO-2026-4945 in go-jose reached through oidc.

Five dependencies — minio-go, minio/mux, etcd client/v3, go-oidc and lestrrat-go/jwx — were deliberately not upgraded. MinIO consumes this module through replace, and Minimal Version Selection chooses the highest version in the entire graph, so upgrading them here would also pull the server forward. None has a reported vulnerability requiring that change.

All three workflows previously asked setup-go for a Go version lower than go.mod required and failed on the first Go command; they are now aligned. The linter also fetched an installer from the master branch and reinstalled it on every run. The URL and version are now pinned to v2.11.3, and a matching installed version skips the download.

Changes Deliberately Not Taken from Upstream

  • AIStor policy vocabulary (Memory/cortex, Tables/Iceberg, KMS, compression and annotations) and the typed action-constant refactor, none of which the community server implements. This is the source of the action-vocabulary delta.
  • securityAuditAdmin, which grants admin:ExportIAM and therefore exposes every secret key despite what the name suggests.
  • rng AVX2/NEON assembly. Revisiting the arm64 half is noted above as a future sync candidate.
  • net.BandwidthBytesPerSec (declared but never read upstream), replicationAdmin and DistJobStatusAction.
  • Two changes initially taken and removed after review: the consolereadonly built-in policy and GetAllGlobalCertificates. Neither has a consumer. Once operators bind a built-in policy name to users, withdrawing it is particularly unsafe: policy mappings persist by name, and an unresolved name merges into an empty policy that denies everything. Its inherited admin:CreateUser Deny also cannot be combined with iamAdmin. The certificate helper inventoried a cache the community server never populates.
  • Upstream’s golangci-lint tool directive, which would add roughly 200 linter dependencies to every downstream consumer’s module graph.

Deliberately Deferred

The general problem in minio/minio#20449 — that bucket/* still reaches ListBucket, GetBucketLocation, the configuration reads, CreateBucket and the three tenant-plausible writes — is not closed here. Closing it means revoking grants real deployments depend on, so it belongs to a release that carries a migration path.

What that release owes operators is more than a longer action list, because no one can enumerate every deployment’s stored policies — which puts a hard ceiling on any approach that picks the protected set by guessing. Three things raise it:

  • A startup policy audit that walks stored policies and names each one whose meaning changes, in both the grant and the deny direction. It is read-only and can ship before the enforcement change rather than with it, turning an upgrade surprise into a pre-upgrade checklist.
  • A denial that explains itself. When a request is refused because only an object-scoped grant matched, say so and name the compatibility switch. A break an operator can diagnose in thirty seconds costs an order of magnitude less than a silent one.
  • A switch with a scope. MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH is all-or-nothing today, so an operator who needs one action back must reopen the self-escalation path along with it.
  • d8b1fa7: fix(policy): settle the bucket-write hardening’s scope and monotonicity
  • 1f97549: fix(policy): extend the bucket-write hardening to every bucket-only write
  • 3c24ad1: fix(policy): withhold object-only grants from sensitive bucket writes
  • da6a22a: docs: say what this fork is and how to depend on it
  • 4055b2f: fix(xtime): marshal Duration as a duration string
  • 13c26cd: fix(rng): initialize the reader subkeys from the seeded entropy
  • 88b37ac: fix(certs): stop file watchers on every exit path
  • 74dd36e: fix(ldap): keep StartTLS when ServerInsecure is also set
  • 424c3d0: fix(ldap): close the connection when StartTLS fails
  • 045d10f: fix(ldap): guard a nil TLS config and arm the StartTLS deadline
  • 5c4bf50: fix(policy): prefer the exact key name over the canonical header form
  • 802539f: chore(deps): refresh the dependency set and declare the real minimum Go
  • e4ec64a: ci: build on the Go version go.mod requires, and prove the declared minimum
  • 747d8b8: build: pin the golangci-lint installer and skip a matching install

2.13 - Silo 20260903 Released

The complete release since 20260806: authenticated replication semantics, per-bucket CORS, serialized bucket metadata, AWS-aligned authorization and checksums, safer SSE-C copies, Go 1.27.1, and the final 20260903 mcli and SILO component line.

Status on 2026-09-13: this remains the latest published Server; its dependencies have not changed. Later security, storage and dependency fixes are on main, including the unreleased SN-2026-011 fix. New standalone mcli/pkg releases do not replace this artifact’s contents; see the component matrix.

Version date: 2026-09-03 · Published: 2026-09-04 · Version: RELEASE.2026-09-03T13-18-01Z · Previous release: RELEASE.2026-08-06T00-00-00Z

SILO 20260903 is the security and correctness release following the first fully rebranded Silo release. Its main theme is not a new delivery surface but a tighter set of server invariants: client headers do not create replication authority, browser origins cannot trigger bucket-metadata I/O before authentication, concurrent bucket-configuration writers cannot silently overwrite one another, and authorization and checksum behavior more closely match AWS S3.

This note deliberately uses 20260806 as its only release baseline. Intermediate component builds remain identifiable in their source repositories, but the public documentation below consolidates the complete server, Console, shared-package, and client result delivered on 2026-09-03.

Highlights

  • Replication trust is authorization-derived. X-Minio-Source-Replication-Request and related internal fields gain replication semantics only after signature verification, an exact marker check, and s3:ReplicateObject or s3:ReplicateDelete authorization. Untrusted fields are stripped after authentication so SigV4 remains valid.
  • Per-bucket CORS is complete. PUT, GET, and DELETE ?cors persist a real bucket policy, override the server-wide fallback, and converge through site replication. The pre-authentication lookup is resident-only and remains fail-closed during startup and after a known metadata-load failure.
  • Bucket metadata updates are serialized. A shared metadata.lock covers all whole-record configuration writers, migrations, imports, site adoption, and healing. ForceCreate no longer erases existing configuration, and an Object Lock document always implies plain Enabled versioning.
  • Authorization matches the requested operation. Explicit version deletion requires s3:DeleteObjectVersion; user and group enable/disable calls require the action matching the requested status; policy writes reject empty ARN namespaces.
  • Checksum behavior moves toward AWS S3. Upload parts can receive server-computed checksums, federated UploadPartCopy returns its checksum, multipart completion reports ChecksumType, invalid assertions use AWS-compatible failures, and CRC64NVME with COMPOSITE is rejected.
  • SSE-C reads and copies authenticate the key consistently. Zero-byte objects and GetObjectAttributes no longer bypass key validation; null-version rewrites and in-place key rotation no longer leave unreadable ciphertext or inconsistent checksum metadata.
  • The release was simplified before publication. Dead code and obsolete lint exemptions were removed, compatibility snapshots now track served surfaces instead of source-level noise, dependencies were pinned to reviewed revisions, and flaky timeout tests received a private random source.
  • The client release is part of the same baseline. mcli 20260903 adds read-only checksum verification, fail-closed credential redaction, repaired JSON metrics and S3 Select behavior, strict policy writes, signed packages, provenance, and verified amd64/arm64 images.

Security fixes

The repository security ledger assigns stable local identifiers to five findings without CVEs. An SN- identifier is not a CVE. The complete threat models, compatibility effects, and operator actions are collected in SILO 20260903 Security Notes.

ID Affected surface Fixed invariant Operator impact
SN-2026-006 SSE-C reads and copies of zero-byte objects The supplied customer key is authenticated even when there is no payload block to decrypt Wrong keys now return 403 AccessDenied; no change for correct keys
SN-2026-007 GetObjectAttributes on SSE-C objects Object attributes require the customer key, except for an actually authorized replication peer A bare replication marker cannot skip key authentication
SN-2026-008 Internal replication fields across reads, writes, multipart, deletes, Snowball, and events Replication semantics require authentication plus the matching replication permission Ordinary clients carrying internal-looking headers keep ordinary S3 semantics
SN-2026-009 Admin user and group status changes Enable and disable are authorized against different actions Split custom admin policies that previously granted only one action but relied on both
SN-2026-010 DeleteObject / DeleteObjects with versionId Explicit version deletion requires s3:DeleteObjectVersion Review grants and add an explicit Deny where Deny s3:DeleteObject was intended to block permanent deletion

All five defects were inherited from the archived upstream server lineage and affect every earlier SILO release. SN-2026-008 completes the receiver-wide repair begun by CVE-2026-34204.

The embedded Console v2.3.0 line also contains a trusted-proxy boundary, TLS verification repair, response and log redaction, and bounded WebSocket forwarding. The standalone Console release and the server embedding remain separately verifiable artifacts.

S3 correctness and interoperability

Multipart checksums

This release closes the multipart interoperability group reported in #46, #47, #48, and #50:

  • UploadPart computes the selected checksum when the client selects an algorithm at upload creation but omits a per-part checksum header.
  • UploadPartCopy returns the remote checksum through a legacy federated backend instead of dropping it.
  • CompleteMultipartUpload includes ChecksumType and uses AWS-compatible error codes when a checksum is absent, malformed, contradictory, or unsupported.
  • Unknown checksum algorithms and the invalid CRC64NVME + COMPOSITE combination are rejected rather than silently canonicalized.

The fixes are deliberately layered. The wire parser rejects invalid declarations, the multipart state records the selected algorithm and type, and completion validates the final claim against that state. Ordinary clients that do not opt into additional checksums are unchanged.

The individual evidence records are server-computed UploadPart checksums, multipart completion error semantics, and ChecksumType propagation.

CopyObject and SSE-C

CopyObject now:

  • calculates checksums over the logical object before optional compression;
  • returns the checksum fields in its response;
  • preserves transform state on metadata-only copies;
  • decrypts source checksum metadata with the source key and rewrites it for the destination key;
  • handles null-version copies in either pool direction without losing current-version metadata;
  • re-encrypts an in-place SSE-C key rotation when the object layer must rewrite the data.

The zero-byte and attributes fixes remove two independent places where “no data was decrypted” had accidentally been treated as “the key was authenticated.” The new rule is explicit: access to SSE-C metadata or ciphertext requires a successfully unsealed object key, unless the request has already crossed the authorized replica boundary.

The CopyObject state matrix and live interoperability evidence are preserved in CopyObject SSE-C, checksum, and response correctness.

Listing and federation

  • ListObjects shortcuts now return NoSuchBucket for a missing bucket even when a prefix selects an early-return path (#32, PR #37).
  • Federated UploadPartCopy preserves the backend checksum in the public response (#72).
  • Site-replication status counts metadata operations per site instead of multiplying a group-wide result, and validates CORS as an independent configuration field.

The missing-bucket shortcut is documented in ListObjects must prove the bucket before it optimizes.

Per-bucket CORS and request trust

Bucket CORS was implemented end to end rather than as a handler-only shim:

  1. the S3 XML type and rule matcher enforce the protocol grammar and multi-rule preflight behavior;
  2. bucket metadata stores the raw configuration and returns it byte-for-byte on GET ?cors;
  3. the outer middleware uses the bucket policy when present and the global origin setting only as fallback;
  4. site replication carries CORS in its own tombstone-aware last-writer-wins register;
  5. pre-authentication lookup never loads metadata or creates cache entries;
  6. startup, a known bucket-metadata load failure, deletion, refresh, and on-demand reload all update the fail-closed state explicitly.

The load-failure state is intentionally retained. A pre-signed URL authenticates itself without consulting bucket policy; if the server forgets that a real bucket’s CORS document failed to load, falling back to a permissive global policy would discard the only browser-origin boundary around that otherwise authorized request.

Replication-request hardening shares the same principle. The server authenticates the original request first because internal headers may be covered by SigV4, derives one private trust decision from the authenticated identity and permission, and only then sanitizes an untrusted request clone. The clone shares the original trailer map so streaming checksums remain visible. Snowball workers derive trust per entry rather than inheriting a request-wide privilege bit.

The full design and rejected alternatives are preserved in No I/O Before Auth, No Privilege From Headers.

Bucket metadata and Object Lock

Bucket configuration is stored as one .metadata.bin record but was previously updated through independent locks for policy, lifecycle, encryption, tags, quota, replication, Object Lock, and CORS. Two valid read-modify-write operations could therefore each succeed while the later save silently discarded the earlier field.

This release introduces one bounded metadata.lock transaction around every whole-record mutation. The same lock covers:

  • normal configuration saves and deletes;
  • legacy metadata migration and import;
  • bucket creation, ForceCreate, and site adoption;
  • healing and inconsistent-version repair;
  • replication receive paths that merge changed fields.

The implementation does not turn the metadata system into a generic transaction framework. It keeps the existing record and parsers, centralizes only the serialization boundary, and narrows replicated updates to fields that actually changed.

Object Lock receives a second invariant. Once a valid lock configuration parses as enabled, versioning is normalized to a plain Enabled document; prefix exclusion and suspended states cannot survive. The decision is based on the parsed configuration, not byte equality with the smallest possible XML document, so a valid Default Retention rule cannot bypass it. Update, read-back, and disk-reload tests cover both malformed prior versioning shapes.

IAM, policy, and configuration behavior

  • Explicit version deletion follows AWS s3:DeleteObjectVersion authorization while replication deletes retain s3:ReplicateDelete.
  • SetUserStatus and SetGroupStatus authorize the requested target state instead of always checking the enable action.
  • New and updated named policies and service-account policies reject bare S3, S3 Tables, and KMS ARN namespace prefixes, their historical serialized forms, and statements that combine Resource with NotResource. Stored policies continue to load.
  • Enabled legacy PostgreSQL or MySQL notification targets must supply a connection string. Startup now fails with a credential-free diagnostic instead of silently discarding all targets.
  • MINIO_CONFIG_ENV_FILE parsing preserves named targets, quoted values, and supported shell-like assignments without evaluating the file as shell code.

These changes are compatibility tightenings, not format migrations. Review the upgrade checklist before deploying.

Detailed records cover target-state user/group authorization, safe configuration environment-file parsing, and fail-fast legacy database notification migration.

Components

The 20260903 dependency and companion-release line is pinned as follows:

  • Go 1.27.1
  • minio-go/v7: upstream-compatible pseudo-version ending in 0e78d3f18efe; the temporary silo-go fork is retired from the server graph
  • madmin-go/v3: v3.0.110
  • silo-pkg/v3: v3.13.2, under its own github.com/pgsty/silo-pkg/v3 module path
  • embedded Console: v2.3.0, including the trusted-proxy, TLS, redaction, and WebSocket security work
  • bundled mcli: RELEASE.2026-09-03T07-13-05Z, with the mc compatibility alias preserved
  • Helm chart: 7.0.2; use the published server image pgsty/silo:RELEASE.2026-09-03T13-18-01Z and the mcli image pgsty/mc:RELEASE.2026-09-03T07-13-05Z

Tagged-source Helm note: the signed Server tag retains the provisional RELEASE.2026-09-03T00-00-00Z value in helm/silo. Post-release commit 4c164907f corrects the current main chart to the published tag. If you render Chart 7.0.2 directly from the tagged source archive, set image.tag=RELEASE.2026-09-03T13-18-01Z explicitly; binary and package artifacts are unaffected.

The server deliberately preserves the MinIO-compatible Go module and wire identifiers. Replacing an import target does not rename the public S3/Admin API, MINIO_* configuration, x-minio-* headers, /minio/* routes, or on-disk metadata.

Engineering cleanup

The final release pass removed complexity where it did not protect a compatibility or correctness boundary:

  • removed dead encryption helpers, obsolete handler paths, and unused event-target functions;
  • replaced a source-level exported-symbol inventory with the smaller served-route compatibility baseline;
  • removed the wait_pipe lint exclusion and migrated to gomodguard_v2;
  • consolidated CORS load-failure lifecycle changes behind two helpers instead of open-coding set operations in six paths;
  • narrowed site-replication imports to changed metadata fields;
  • gave dynamic-timeout tests their own random source so parallel tests cannot mutate a package-global seed;
  • retained the shared metadata lock, CORS tombstones, two-level replication trust, and adversarial tests because each protects a reproduced failure rather than a hypothetical abstraction.

The complete adversarial review, including the fixes found after the first “ready” claim, is recorded in SILO Server 20260903 Pre-release Review.

Upgrade checklist

Read Upgrading from RELEASE.2026-08-06 before changing production. At minimum:

  1. grant s3:DeleteObjectVersion to identities that intentionally delete explicit versions; add its Deny next to any Deny s3:DeleteObject intended to prevent permanent deletion;
  2. split custom admin grants for enable and disable operations;
  3. correct bare ARN prefixes before re-submitting policies;
  4. add connection strings to enabled legacy PostgreSQL/MySQL notification targets;
  5. verify applications that explicitly choose checksum algorithms or types;
  6. upgrade every member of a site-replication group before creating or changing per-bucket CORS;
  7. snapshot CORS configuration before rollback, because 20260806 does not understand it;
  8. switch every node of a distributed cluster to the same binary in one maintenance operation; mixed Silo binaries do not form a cluster.

The on-disk object and erasure formats are unchanged. This does not make a mixed-version cluster supported.

Known issues and explicit deferrals

These items are not fixed by this release:

  • Conditional delete (#10, PR #12). DeleteObject ignores HTTP If-Match and DeleteObjects ignores each <Object><ETag>. Both delete unconditionally. A reviewed single-object repair exists outside this release, but batch semantics and the original PR are incomplete; shipping a partial condition contract would be riskier than the documented deferral.
  • Non-CORS multi-site configuration deletion (#77). A peer can restore a policy, SSE, tag, or quota configuration deleted on another site. Single-site deployments are unaffected. CORS is not affected because it uses a tombstone-aware register. Deployments relying on these replicated deletions must verify every site.
  • ListMultipartUploads filters (#79). prefix behaves like an exact key match, while max-uploads, key-marker, and delimiter are not fully honored; see the conformance analysis.
  • Legacy federation CopyObject (#99, #100). The legacy federated backend can ignore a requested checksum algorithm and rejects inline source objects. Deployments not using that backend are unaffected.
  • Mixed-version site replication. A 20260806 peer accepts but ignores bucket CORS and continues reporting a mismatch. Do not configure CORS until all sites are upgraded.
  • Rollback of bucket CORS. 20260806 drops the unknown CORS field if it rewrites the bucket record. Export the configuration first and recreate it after upgrading again.

Open enhancement work such as ILM relocation (PR #60), broader SSE support (#61), operator discovery (#30), NATS target hot reload (#40), and Renovate (#20) is outside this release’s production-safety boundary.

Verification and evidence boundary

The complete local acceptance was run on ebac0ca73bbf251b070bb6df4d8005015841f901:

  • full cmd and internal test suites;
  • go test -race ./cmd, passing in 365.448 seconds;
  • lint with 0 issues, the rebrand/compatibility guard, generated-file checks, and govulncheck;
  • make verify across FS, erasure, distributed erasure, erasure sets, multi-pool, and IPv6 multi-pool: 174 PASS / 0 FAIL.

One subsequent one-line fix, 84e1580a4, clears the CORS load-failure bit after a successful on-demand GetConfig reload. Later dependency work moves the toolchain to Go 1.27.1 and x/crypto 0.56.0. On exact release source 9b11dc9469e650815b775cb47b039610644f5da4, git diff --check, full cmd and internal tests, the complete cmd race suite, rebrand and generated-file guards, lint, govulncheck, and remote Go CI, VulnCheck, Test Release, and Release workflows pass.

The final functional acceptance used four Ubuntu 24.04 arm64 VMs with four XFS data disks each. It covered a coordinated 20260806 → 20260903 upgrade, rollback, and second upgrade; the complete mcli functional script over TLS; 1,004-object upload/download checksum comparison; one-node and quorum-loss behavior; single-drive reconstruction; and two-site replication of objects, versions, CORS, tags, quota, ILM, Object Lock, IAM, delete markers, and offline catch-up. The cluster finished with all 16 drives online and no unrecovered data mismatch.

Container publication exposed one delivery-only pin error after the binary/package Release was public: the first Docker workflow stopped before pushing any image because the tagged Dockerfile selected mcli 20260903 but retained the two 20260901 archive digests. Commits ce2326c94 and 9936a69d8 corrected those pins and added a narrow recovery mode that resolves them from the immutable, attested mcli checksum manifest. Recovery run 33847571968 then built, ran, SBOM-attested, and published both architectures from the original Server release archives. The classic release/latest manifest is sha256:b616a0cf8cb281e7e6bb3c9b1fb53875b4016a2878223925541c18f82d6c5ca3; distroless is sha256:2711cdc5bce0ada353a4871dcda60f48f46414fdd69a45c36c71f6b7ebe1aead.

Delivery remains independently checkable rather than inferred from those source tests. The published server GitHub release identifies the exact tagged source and verified archives, packages, checksums, SBOMs, and provenance; its classic and distroless images publish matching amd64/arm64 manifests. The separately published mcli release is immutable, contains 19 verified assets, and publishes matching amd64/arm64 release and latest images. Shared package v3.13.2 and Console v2.3.0 retain their own release evidence.

Resolved issue ledger

The tracker was reconciled during release preparation. Core issues are closed; deliberately excluded work remains in its own open issue:

Tracker Resolution
#102 Closed by merged PR #103: shared metadata.lock and complete writer/migration/adoption coverage; reproduction-first residual audit split to #105
#58 Closed by merged PR #104: explicit-version authorization with multi-delete context preservation and least-privilege replication tests; PR #59 closed as superseded
#46, #47, #48, #50 Multipart checksum selection, computation, response, validation, and errors
#32, PR #37 Missing-bucket listing shortcut returns NoSuchBucket
PR #57 CompleteMultipartUploadResult.ChecksumType is present in the selected client/server stack

Selected changes

  • 1c9a2431f through 13e6458d9: implement per-bucket CORS and site replication
  • 938603458 through 04b097fd9, plus Snowball follow-ups: authenticate replication semantics and remove pre-authentication metadata I/O
  • f9f9fa6c9 through 32a1b81e4: reproduce and serialize cross-type bucket metadata updates
  • 3b5de82f5, 21646eebd: enforce Object Lock’s versioning invariant, including retention-rule documents
  • f8b598f1d through d2d47a41f: align explicit-version delete authorization
  • b73581b05, 474cd5801, 74c97d005: authenticate SSE-C zero-byte and attributes reads
  • 7fea6d5a5, 5d152416d, 7e079ff05, d28885d0e: align multipart checksum behavior
  • c0e715977, e73436c99, ffb70eb37: repair CopyObject transforms, checksums, and SSE-C rotation
  • 84e1580a4: clear the load-failure guard after an on-demand metadata reload succeeds

Acknowledgments

This release incorporates reports and proposed fixes from community contributors as well as the fork’s security and compatibility audit. In particular, the checksum, missing-bucket listing, and explicit-version authorization work started from public issues and pull requests. The release’s final shape also reflects adversarial review: findings were reproduced before repair, inherited limitations were separated from release regressions, and validation claims were rerun against the tree they describe.

The complete authorship record remains in CONTRIBUTORS.md.

2.14 - Explicit Version Deletes Now Require DeleteObjectVersion

SILO now selects DeleteObject or DeleteObjectVersion from each request’s effective version ID, while preserving the established least-privilege replication target policy.

Release status: this change is implemented in pgsty/silo#104, tracking issue #58. Publishing this note does not by itself mean that a server release, package, image, or deployment contains the change.

SILO now maps object-delete authorization to the operation the request will actually perform:

Request Required action
No versionId s3:DeleteObject
Version UUID s3:DeleteObjectVersion
Explicit versionId=null s3:DeleteObjectVersion
DeleteObjects The mapping is applied independently to every XML entry

Previously, SILO required s3:DeleteObject for every case and used s3:DeleteObjectVersion only as an explicit-deny check. A principal holding only DeleteObject could therefore permanently remove a named historical version. Conversely, a least-privilege purge principal holding only DeleteObjectVersion could not perform the operation it was intended for.

After this change, a DeleteObject-only principal can still perform an unversioned delete or create a delete marker, but receives AccessDenied for a named UUID or null version. A DeleteObjectVersion-only principal can remove the named version but cannot create a delete marker. Explicit denies and s3:versionid conditions retain normal policy precedence. Multi-delete uses each entry’s VersionId; a query-level decoy cannot change another entry’s condition value.

Replication compatibility

Bucket and site replication target policies do not need to add s3:DeleteObjectVersion. An authenticated request earns replication delete trust through an exact internal marker and s3:ReplicateDelete; the receiver then preserves the deployed s3:DeleteObject + s3:ReplicateDelete contract. An explicit deny on s3:DeleteObjectVersion continues to block a replicated version purge.

This distinction prevents an upgraded target from silently rejecting permanent delete replication and also avoids granting new delete capability to a ReplicateDelete-only credential. A real two-site regression used a target user with the documented minimal policy and no DeleteObjectVersion; both a permanent version delete and a delete marker converged successfully.

Upgrade impact

  • Review user, service-account, OPA, and external authorization policies that currently grant only s3:DeleteObject but perform mc rm --version-id, mc rm --versions, Console delete-all-versions, or SDK deletes with versionId.
  • External authorization plugins now see one s3:DeleteObjectVersion decision for an ordinary named-version delete instead of the former deny-only check followed by s3:DeleteObject.
  • X-Minio-Force-Delete prefix cleanup remains gated by s3:DeleteObject; it is not an explicit-version S3 request.
  • There is no wire or storage-format migration. Rolling back restores the old authorization mapping but does not alter stored objects or metadata.

See Object Deletion for the operator-facing permission matrix.

2.15 - mcli 20260903 Released

The complete client release since 20260806: Go 1.27.1, silo-pkg 3.13.2, read-only checksum verification, repaired JSON metrics and S3 Select, fail-closed credential redaction, strict policy writes, private support artifacts, signed packages, provenance, immutable releases, and verified multi-architecture images.

Published: 2026-09-03 · Version: RELEASE.2026-09-03T07-13-05Z · Source: a2ef95c0

Twenty-eight days after mcli 20260806, this release carries everything the client accumulated since. The toolchain moves to Go 1.27.1 and the shared dependency stack is aligned with the SILO family, with silo-pkg imported under its own module path so go.mod says what the binary is built from. The first genuinely new command since the fork arrives — mcli checksum verify, a read-only audit of stored S3 additional checksums against the bytes an endpoint actually returns. The panic that made --json admin prometheus metrics unusable is fixed, as is the response double-close that could panic after mcli sql had printed valid rows. And every path that can print a credential — --debug traces, admin trace, error messages, JSON error documents, checksum reports — was rewritten fail-closed after adversarial review. The release process changed as much as the code: artifacts come from a GPG-signed tag at the exact tip of main with green CI for that commit, ship LICENSE, NOTICE, and a generated CREDITS in every format, carry Sigstore build provenance, and land in an immutable GitHub release.

The release was built from a2ef95c0, the merge of pgsty/mc#27: 58 reachable commits, 50 of them non-merge commits, across 262 files (+12,742 / −6,566 lines) since RELEASE.2026-08-06T00-00-00Z. The intermediate 20260901 build remains available on GitHub, but this page treats 20260806 as the public documentation baseline and records the complete 20260903 result.

Warning

Behavior changes

  • macOS 13 is the new floor. Go 1.27 links darwin binaries with a minimum OS version of 13.0; the darwin archives no longer run on macOS 12 and earlier.
  • mcli --json admin prometheus metrics works again and returns metric families in deterministic name order. Plain-text metrics output is unchanged.
  • pipe with empty standard input now performs a regular zero-byte PUT instead of a one-part multipart upload. The resulting ETag is d41d8cd98f00b204e9800998ecf8427e rather than the <md5>-1 form. Non-empty input is unchanged.
  • Policy writes are validated strictly on the clientadmin policy create, admin user svcacct add/set, idp ldap accesskey create/edit. A bare ARN such as arn:aws:s3::: is rejected (write arn:aws:s3:::* to mean every resource under that prefix); a named policy without Version is rejected; an empty policy is rejected; mixing S3 and admin actions in one statement is rejected. The last three were already refused by the server, so only bare ARNs change what can be stored. Reads of existing policies remain permissive. Session policies for service accounts may still omit Version, and admin user svcacct set --policy still accepts an empty document — that is the way to clear an inline policy and return the account to its inherited one.
  • A malformed global flag before the command name is now an error. mcli -H "bad" ls, --resolve and --limit-* values that fail to parse at the app level used to be silently ignored and the command ran without them. They now fail with the same message the command-level form produces, reported once on stderr — or as a JSON error document under --json — where the CLI library used to echo the text to stdout first.
  • Error messages no longer echo secrets. alias set reports only the length requirement for a bad secret key; MC_CONFIG_ENV_FILE parse errors name a line number instead of repeating the line; SSE-C key errors withhold the key; admin config set masks secret-bearing key=value pairs; remote replication targets are printed without credentials; a malformed --custom-header entry is reported by position, never by value.
  • Support artifacts are private. Files written by support diag, support profile, support perf, support inspect, admin cluster bucket export, and admin cluster iam export are created with mode 0600, including the rotated backup of a previous file. They were 0644 or 0666 before.
  • --debug shows less. An Authorization header keeps only its scheme (AWS4-HMAC-SHA256 **REDACTED**); tokens, cookies, API keys, SSE-C keys and every --custom-header value are redacted; 2xx response bodies are no longer dumped. admin trace output is redacted the same way in both its default and its verbose rendering: header values by name, query strings, bodies, error text and annotations.
  • The User-Agent reports minio-go/v7.3.0 (the SDK’s version constant) instead of minio-go/v7.0.98.

Major Changes

  • Go 1.27.1 across the module, container build stage, Linux/macOS/Windows CI, cross-builds, tidy checks, and vulnerability analysis. The Go-maintained module family is current: x/net 0.58.0, x/sys 0.47.0, x/term 0.45.0, x/text 0.41.0, x/crypto 0.56.0, x/mod 0.40.0, x/sync 0.22.0, and x/tools 0.49.0. golangci-lint stays outside the product module graph and is installed independently at the pinned v2.13.1.
  • The SILO package under its own module path. go.mod now requires github.com/pgsty/silo-pkg/v3 v3.13.2 directly. The client’s policy validation depends on helpers that exist only there — policy.ParseConfigStrict and policy.Resource.IsBareARN — and a replace of upstream minio/pkg would not be inherited by projects consuming this module (pgsty/mc#12), so the requirement is declared honestly. Upstream minio/pkg survives only as an indirect dependency of colorjson, held at v3.6.1 by the dependency-floor check, and gomodguard blocks the upstream import path from returning. minio-go moves from v7.0.99 to the latest upstream minio/minio-go master commit, represented by v7.3.1-0.20260828014306-0e78d3f18efe; the interim pgsty/silo-go fork is retired. The final refresh includes etcd 3.7.1, go-pretty 6.8.3, klauspost/compress 1.20.0, Prometheus client/common/procfs 1.24.1/0.71.0/0.22.0, grpc-gateway 2.30.0, gRPC 1.83.2, Protobuf 1.36.12, and JWX 3.2.0. go-systemd deliberately stays at 22.6.0 through SILO’s portability pin — 22.7.0 does not compile on NetBSD.
  • Runtime version floors are guarded. A first dependency pass let the removal of development tools silently lower a dozen runtime selections — among them shoenig/go-m1cpu, whose older releases crash during initialization on newer Apple Silicon when CGO is enabled — and was rejected. buildscripts/check-dependency-floors now compares every requirement in go.mod against the previous RELEASE.* tag and fails the build on a regression; the only permitted exception is the go-systemd pin.
  • mcli checksum verify (design). Verifies stored S3 additional checksums — CRC32, CRC32C, CRC64NVME, SHA1, SHA256, of type FULL_OBJECT — against the logical object bytes returned by the object API, for one object, a prefix with --recursive, all versions with --versions or an exact --version-id, or a JSON Lines candidate manifest. It reads with If-Match on unversioned objects and re-stats afterwards, so a write that races the verification is reported as UNKNOWN_OBJECT_CHANGED rather than as a mismatch. Each object yields one of MATCH, MISMATCH, NO_CHECKSUM, WOULD_VERIFY (dry run), one of ten UNKNOWN_* classifications, or one of three SKIPPED_* results; the summary carries objects, a verified count (MATCH + MISMATCH, the only results that actually recomputed a checksum), per-status counts and incomplete. --fail-on accepts mismatch, unknown, no-checksum (fails when any object lacks a checksum or when nothing was verified at all), any (default: mismatches, unknowns, and objects skipped by --max-size), or none. --report writes the same records to a new 0600 JSON Lines file. Results go to stdout whether or not stdout is a terminal, and with --json they are compact JSON Lines outside a terminal — the non-TTY silence reported as pgsty/mc#5 is fixed. Dry runs, --max-workers, --max-size, time filters, and SSE-C through --enc-c are supported. The command is verified against a real SILO server in CI.
  • Credential redaction, fail-closed. Three rounds of adversarial review by Codex showed that the original redaction — and its first two rewrites — recognized the shapes it expected and let everything else through: access keys containing a slash, an Authorization value whose first field is empty so the second one hides, non-hex signatures, secrets tucked into the credential scope or SignedHeaders, Proxy-Authorization, cookies, X-Api-Key, response trailers, a redirect Location with userinfo, a server that reflects request headers into an error body, a JSON-escaped secret inside --json error output, a short token that appears twice. The final design has three rules and no exceptions. An Authorization-class value keeps only its scheme. A header whose name looks like a secret, or that the caller supplied through --custom-header, is redacted without inspecting its content. Text the client did not generate itself — response bodies, server error messages, trace notes — is scanned for credential shapes and then scrubbed of every secret literal the process has learned. That last layer is backed by a process-wide secret registry: alias secret keys and session tokens, MC_HOST_* values, SSE-C keys in every encoding, tier and service-principal secrets, GCS credential files, LDAP passwords and the STS credentials they return, SUBNET tokens, --secret-key/--account-key/--api-key flags, and secret-bearing key=value arguments are registered as they are read, before validation and before any network call. Occurrences are collected as intervals and merged before replacement; JSON error documents are scrubbed on the decoded value tree, string leaves only, so keys and compact non-terminal formatting survive. Forty-odd adversarial cases, a black-box test that runs the real client against an endpoint that reflects everything it receives, and live captures against a SILO server pin the result.
  • admin trace redaction. Server-supplied header maps for other clients’ requests take a path the S3 tracer never touched. Both the human-readable and the JSON rendering, in the default and the verbose form, are now produced from redacted copies — the event is shared between them and never mutated. Query strings (a presigned URL’s signature, a session token in the first parameter), bodies, error text and annotations are scrubbed with the event’s own header values, so a token another client sent cannot reach privileged trace output. Headers are redacted by name: a --custom-header another client sent under a name that does not look like a secret is shown as the server saw it.

Fixes

  • JSON metrics no longer panic. mcli --json admin prometheus metrics failed for both the v2 and v3 metrics APIs with panic: Invalid name validation scheme requested: unset in the 20260806 client. madmin-go built the zero value of the Prometheus text parser, which prometheus/common 0.66 and newer reject. The client now constructs the parser with UTF-8 validation, returns ordinary parse errors instead of panicking, and emits families in name order.
  • pipe with empty input performs a regular PUT; the one-part multipart upload it used to issue for zero bytes was needless and produced a multipart-style ETag.
  • RPM payload validation aligned with the signing script, so an RPM whose contents do not match the release metadata is refused before it is signed.
  • sql no longer panics after printing its rows. The SDK’s select reader closes the HTTP response itself on every terminal event, and the client closed it a second time; on a zstd-compressed response the two drains raced and about two in five queries against a SILO server ended in panic: invalid memory address with exit status 2 after the rows had already been written (pgsty/mc#23). The client now cancels the request and drains the result pipe instead of closing the response again; 820 consecutive queries on a live server ran without a panic where the previous client failed 21 of 50.

Hardening

  • Operator-produced files are no longer world-readable. support diag warned that its archive may contain sensitive environment information and then wrote it 0666; the shared moveFile helper relaxed every 0600 temporary file to 0644 on its way to the final name, and a 0644 left by an earlier client was carried into the rotated backup even when source permissions were preserved. Every support artifact is now created 0600 and chmodded before the first byte is written.
  • Test_FullSuite fails on panic. The functional suite recovered from panics to run cleanup and then reported success; a panic anywhere in the suite, including in the binary build, was invisible to CI.
  • A leftover play/ alias in the new command’s help — the only file in the repository still advertising MinIO’s operated demo alias — is replaced, and the brand gate now rejects bare aliases in command examples, not only hostnames.
  • AGPL notices completed on six fork-authored files that carried only the first half of the license text.

Engineering and Delivery

  • Third-party notices in every artifact. CREDITS is generated by buildscripts/gen-credits.sh from the union of the runtime modules linked into all six GoReleaser targets and is compared in CI so a dependency change cannot leave it stale. LICENSE, NOTICE, and CREDITS ship in the GoReleaser archives, the DEB/RPM/APK packages, and every container image under /licenses/.
  • CI runs on main. Go and Crosscompile ran only on pull requests, so the merge commit — the one that gets tagged — was never tested; both now run on every push to main. Their concurrency group was keyed on github.head_ref, which is empty outside pull requests, so every manual run cancelled every other; it is now keyed on the pull request number with the ref as fallback. VulnCheck also runs weekly, because a fork of an archived upstream can go weeks without a commit. The functional suite in CI now targets SILO RELEASE.2026-08-06T00-00-00Z.
  • Release only from signed tags. The artifact workflow has no manual trigger; artifacts come from a pushed RELEASE.* tag, and the event SHA, the commit the tag points to, and the checkout must all agree. verify-release-tag.sh requires an annotated tag with a valid OpenPGP signature from the expected key — fingerprint and key location come from repository variables rather than from the tree the tag points at, and the imported key must match the expected fingerprint. check-release-commit.sh requires the tagged commit to be the exact tip of origin/main and demands a successful push-to-main run of four workflows for that SHA, matched by workflow file path; pull request and manual runs are not evidence. check-release-state.sh (pgsty/mc#6) runs serially per tag, replaces exactly one existing Draft from scratch, refuses a published release, and re-checks for a single Draft before packages are uploaded. Every one of these decisions lives in a shell script with fixture coverage; the fake gh in the tests pins both the verdicts and the arguments passed, verbatim.
  • Provenance and immutability. Archives, checksums, DEB and APK packages carry Sigstore build provenance (gh attestation verify <file> --repo pgsty/mc). RPMs are excluded on purpose: sign-release-rpms.sh rewrites their bytes with a detached GPG signature from key 9592A7BC7A682E7333376E09E7935D8DB9BD8B20 while the release is still a Draft, and an attestation made earlier would name a digest no published RPM has. GitHub releases are immutable once published; fixing an asset means a new tag. The pgsty/mc container image is built only from attested archives, after gh attestation verify binds each one to release.yml@refs/tags/<tag>, the release commit digest, GitHub-hosted runners, its own filename and its SHA-256; the unpacked binary’s ELF machine type must match the archive’s architecture; each per-architecture image is pulled, run, and checked for the tag before the manifest is assembled, and latest is written only from the two-architecture manifest. The UBI base image is pinned by digest and kept current by Dependabot.
  • Repository governance matches a single-maintainer project. Direct pushes to main are allowed; the stale pull-request ruleset and its exact-name status-check coupling were removed after this release. Release integrity does not depend on that branch rule: the artifact gate still requires successful push-to-main Go, Crosscompile, VulnCheck, and Test Release runs for the exact tagged commit. RELEASE.* tags remain administrator-created and immutable; secret scanning, push protection, Dependabot alerts, and security updates remain enabled.
  • Verified before and after tagging. On exact source a2ef95c035d9ae7cc01469a63926900f1786f9e2, tidy/verify, dependency floors, unit tests, the full race suite, lint, actionlint, source-aware govulncheck, and nine standard cross-builds passed. Claude Fable 5.1 at xhigh independently returned GO with no blocking findings after a clean-cache Go proxy/sumdb build, 172 functional checks against a temporary SILO server, a 47-command black-box comparison with the published 20260901 binary, and seven additional cross targets. All four exact-SHA main workflows and both publication workflows passed. The immutable public release contains 19 assets; archives and sidecars were checksum-verified, all 11 attested subjects were bound to the tag and source commit, both RPMs were GPG-verified, and the pgsty/mc release and latest images were independently confirmed as matching amd64/arm64 manifests. The source scan reports no reachable vulnerability.

Compatibility

The full --help tree of this release was diffed against the 20260806 binary: the only difference is the new checksum command. Side by side on the same server, head, cat, get, cp, mirror, find, share, sql, stat, and ls produce identical output and JSON field sets; twenty real-world policy shapes are accepted or rejected identically except for bare ARNs; admin policy info --json is byte-identical. The mc command name and the mcli package name; the ~/.mc / ~/.mcli configuration directories; the github.com/minio/mc module path and all import paths; MC_* environment variables; protocol headers (x-minio-*) and the minio-go user-agent prefix; the minio tier type; the .part.minio transfer suffix; the minio-job scrape job name; and the package formats, asset naming, and YYYYMMDDHHMMSS.0.0 version scheme are unchanged. The client remains compatible with MinIO servers and other S3-compatible endpoints. Projects that consume this module as a Go dependency now resolve github.com/pgsty/silo-pkg/v3 directly and need no replace of their own for it.

Note

Known issues and boundaries

  • checksum verify reports UNKNOWN_READ_ERROR, not MISMATCH, when the server’s own integrity checks refuse a read — SILO’s bitrot protection rejects a corrupted shard before the object API returns it. A MISMATCH means the stored checksum describes different bytes than the object API returns, which is the historical write-path defect the command exists to find; an UNKNOWN is never reported as a MATCH.
  • Access key IDs are redacted wherever they appear inside a credential structure, but a server that reflects a bare access key into an error message will still have it printed: it doubles as the user identifier in legitimate messages. Secrets shorter than three characters are not registered. When the secret key equals the access key — the default minioadmin:minioadmin — the identifier is redacted from error text too, because the string is the secret.
  • --json error documents list the keys of cause.error alphabetically rather than in struct order; the keys and values are unchanged.
  • mcli sql reports a failed query on stderr but still exits 0 (pgsty/mc#25); per-object errors are reported without failing the run so a recursive query can continue. Unchanged from upstream.
  • govulncheck traces GO-2026-5932 in the indirect dependency x/crypto/openpgp; no fixed version exists and the client does not call it.
  • JWX 3.2 enforces JWT crit processing more strictly, as required by the standard. The terminal-width refresh can also adjust alignment for Indic or ZWJ-heavy text; neither change affected the tested command surface.
  • minio/mc#5139 continues to apply: mirror --remove --watch can delete a live object from the target when a non-current version of it is removed from the source.
  • Quality debt that does not block the release — archive reproducibility, test-tool portability, deferred coverage of KMS / SSE-S3 / multi-site replication — is tracked in pgsty/mc#21.
  • 98a8b0f: build: prepare the 20260903 dependency release
  • 6686b64: chore(deps): align the SILO Go dependency stack
  • 423490d: fix: stop JSON metrics from panicking
  • 14b05e6: build: generate and package third-party notices
  • e938cea: fix: align RPM payload signing validation
  • 8158605: release: adopt Silo Go SDK v7.3.1
  • d6ba236: ci: allow manual release workflow runs
  • b6b2655: ci: release an existing tag on dispatch
  • 70a2950: fix: preserve embeddable pkg compatibility
  • aa07b1c: feat: add read-only object checksum verification
  • 1b39423: fix: preserve checksum output in pipelines
  • a31a42b: docs: record checksum verify output contract
  • 2121081: fix: validate policy writes strictly
  • 712bf3a: fix: use regular PUT for empty pipe input
  • 9a3805d: ci: make release retries tag-idempotent
  • 7bc9843: test: make report permissions portable
  • 2d5abc2: chore: complete the AGPL notice on fork-authored files
  • f5c5dc9: build: import the SILO package under its own module path
  • a8e7eb4: ci: test main on push and stop dispatch runs cancelling each other
  • b75d28b: fix: stop checksum verify help from advertising the play alias
  • c05da7b: chore: drop configuration for bots that are not installed
  • a29b455: fix: stop –debug from printing credentials
  • 08244df: fix: keep secrets out of error messages and traces
  • 30a7ea0: fix: stop writing support artifacts world-readable
  • 14bd724: test: fail the full suite when it panics
  • 4abbb3a: feat: expose how much checksum verify actually verified
  • f484d0d: ci: make a release prove its commit, and pin what it builds from
  • 2c33a4e: chore: ignore the local consumer-module review scratch directory
  • bdc742a: fix: close the remaining –debug credential leaks
  • 3998392: fix: withhold SSE-C keys and credential URLs the parser rejects
  • c1358c4: fix: force 0600 on rotated support artifacts too
  • 45e5144: fix: fail –fail-on no-checksum when nothing was verified
  • 3674933: ci: make the release gate provable and its claims true
  • 8b7b2e9: test: exercise checksum verify against a real server
  • 4b36b0a: fix: make credential redaction fail-closed and scrub final output
  • a6c14da: ci: verify build provenance before repackaging archives into images
  • 915cfd9: fix: reject a malformed app-level global flag instead of ignoring it
  • 81e2767: fix: rebuild SigV4 traces from known fields and widen secret scrubbing
  • 625713c: fix: keep only the scheme of a signed header and scrub JSON structurally
  • ec12b64: fix: redact admin trace output and register every credential ingress
  • 1445e88: ci: release only from a signed tag at the tip of main, verify exact provenance
  • f188939: fix: close final credential and release verification gaps
  • f5b88dd: fix: redact the default admin trace and stop the shape sweep mangling prose
  • 59989f8: fix: let svcacct set clear a policy again, report bad global flags once
  • d38412a: fix: classify an SSE-C refusal with a key as a read error, keep policy and size messages clear
  • b291b53: fix: register a URL password as written, keep listing parameters visible in traces
  • 9dfca8f: test: drive the SSE-C refusal classification through the verify path
  • 76b2e89: fix: keep an unquoted DSN password whole and match auth schemes case-insensitively
  • 328efee: fix: register the password before the first separator of an unquoted DSN payload too
  • 7057705: fix: avoid double-closing S3 Select responses

2.16 - mcli 20260806 Released

The client adopts the Silo identity end to end: every MinIO SUBNET path is disabled at build time, the embedded vendor diagnostics-encryption key is gone, and contributions move to a no-CLA, DCO-required policy.

Published: 2026-08-06 · Version: RELEASE.2026-08-06T00-00-00Z

Two days after mcli 20260804, this release completes the client’s transition to the Silo identity. It is deliberately a pure rebranding and lockdown release: --version and --help now present the Silo client, every remaining path to MinIO’s SUBNET service is disabled at build time, the embedded vendor encryption key is removed from the diagnostics tooling, and the contribution policy moves to no-CLA with a mandatory DCO sign-off. There are no dependency changes and no protocol changes in this cycle — go.mod is byte-for-byte identical to 20260804 — so the regression surface is confined to text, command gating, and CI.

Warning

Behavior changes

Every path that previously reached MinIO SUBNET is now disabled at build time and cannot be re-enabled at runtime:

  • mcli license register, mcli support upload, mcli support proxy set, mcli support callhome enable, and the online-renewal form of mcli license update ALIAS print a stable notice — “MinIO SUBNET services (registration, licensing, uploads) are disabled in this Silo build of mc; diagnostics remain available locally.” — and always exit 1. Drop these calls from scripts. The file-based mcli license update ALIAS license.key still works, with the license parsed offline against the bundled public key.
  • mcli support diag / perf / profile / inspect always operate in local (airgap) mode: reports, profiles, and inspect archives are written to local files and nothing is uploaded anywhere. The --airgap flag is still accepted for compatibility and is effectively always on. SUBNET registration is no longer a prerequisite for any of them.
  • mcli support callhome disable|status, mcli support proxy show|remove, mcli license info, and mcli license unregister keep working — they only read or clear local and server-side configuration.
  • Fresh configurations no longer seed the play alias pointing at MinIO’s public demo cluster; the defaults are now local, s3, and gcs. Existing configuration files are never modified, and legacy-config migration still recognizes the historical entries.
  • mcli --version gains an identity line (“Silo object storage client, based on MinIO technology”) and a second copyright line. The first line’s machine-readable format is unchanged, so scripts parsing it are unaffected.

Major Changes

  • Silo identity across the CLI: the client introduces itself as “Silo client for object storage and filesystems”. Roughly 220 help texts were reworked: usage lines that refer to the managed server now say “Silo/MinIO server”, example aliases moved from myminio/play to mysilo, example LDAP DNs moved to dc=example,dc=com, and example tier names to SILOTIER-*. Factual references stay factual: the minio tier type, protocol headers, and third-party interop mentions are untouched.
  • SUBNET disabled at build time: connectivity is compiled out behind a single guard, and the one HTTP choke point that every SUBNET request funnels through refuses with the stable error above. Command entry points gate early, diagnostics force local mode, and the AGPL license notice shown by mcli license info no longer carries a commercial-subscription pitch. A dedicated regression suite (cmd/subnet-disabled_test.go) pins all of this, so an upstream merge cannot silently reconnect anything.
  • Governance — no CLA, DCO required: contributions are accepted inbound=outbound under AGPL-3.0-or-later; contributors keep their copyright, and the maintainers collect no rights beyond the project license. Every commit must carry a Signed-off-by trailer, enforced by a new CI workflow that matches the trailer against the commit author’s email and exempts only GitHub-issued bot addresses. CONTRIBUTING.md, the PR template, and both READMEs document the policy, and the code-of-conduct contact now points at the fork’s maintainer.
  • Dual copyright attribution: runtime output and help now credit both lineages — Copyright (c) 2015-2025 MinIO, Inc. and Copyright (c) 2025-2026 PGSTY — with source builds computing the end year dynamically. NOTICE states the fork relationship, and the non-affiliation with MinIO, Inc., explicitly.
  • Release line renamed to main: workflow branch filters, documentation, and contributor instructions now target main; the legacy master references are gone.

Hardening

  • Vendor encryption key removed: mcli support inspect used to fall back to encrypting its output with an embedded MinIO RSA public key whenever no key was supplied — producing archives only the vendor could decrypt. The embedded key is gone: inspect now relies on the server-generated per-request key that is printed to the caller (or an operator-supplied key), and any encrypted-upload path with no configured recipient fails closed instead of silently borrowing a third-party key. Diagnostic output an operator produces is now always decryptable by that operator.
  • Brand-policy gate: buildscripts/check-branding.sh runs in make verifiers and in CI. It fails the build if MinIO-operated endpoints, commercial upsell URLs, the upstream product identity, or any embedded MII… public key reappear in the command tree — while explicitly allowlisting the preserved compatibility identifiers (environment variables, protocol headers, module path, legacy-migration defaults, and original copyright headers).

Engineering and Delivery

  • CI moved to the Node 24 Actions line: actions/checkout v7, actions/setup-go v7, goreleaser-action v7, and the Docker action family — all still pinned to commit SHAs, with dependabot keeping the pins current.
  • Functional tests run against controlled servers only: the suite defaults to a local server (localhost:9000) instead of MinIO’s public demo cluster, and CI downloads the pinned SILO server release RELEASE.2026-08-04T00-00-00Z from pgsty/silo, verified by SHA-256, before running the suite.
  • Zero dependency changes: no module updates this cycle; the 20260804 security baseline (Go 1.26.5, zero known reachable vulnerabilities) carries over unchanged.
  • Audited before tagging: the release was gated by an independent adversarial review — a full read of the 245-file diff, brand/compatibility grep sweeps, call-graph verification that no command or flag combination can reach subnet.min.io / play.min.io / dl.min.io, and smoke tests confirming every disabled path returns its stable error with exit status 1.

Compatibility

Everything scripts and integrations depend on is deliberately unchanged: the mc command name and the mcli package/binary name; the ~/.mc / ~/.mcli configuration directories (derived from the invoked name); the github.com/minio/mc module path and all import paths; MC_* environment variables; protocol headers (x-minio-*) and the minio-go SDK user-agent prefix; the minio tier type; the .part.minio transfer suffix; the minio-job Prometheus scrape job name; and the package formats, asset naming, and YYYYMMDDHHMMSS.0.0 version scheme. The client remains fully compatible with MinIO servers and other S3-compatible endpoints.

Note

Known issues

The mcli watch regression flagged in the 20260804 notes is resolved on the server side: the fix shipped in SILO 20260804, and this client’s CI now runs the functional suite — including watch — against exactly that release. Pair mcli with SILO server 20260804 or newer to receive bucket events; older published servers remain affected.

Unfixed upstream defects continue to apply, most seriously minio/mc#5139: mirror --remove --watch can delete a live object from the target when a non-current version of it is removed from the source. Exercise caution combining --remove --watch on versioned buckets.

  • 8a883ca: ci: move the branch filters to main and fetch the server from pgsty/silo
  • 8c304dd: ci: move the pinned actions onto the Node 24 runtime
  • 02b1c11: docs: name the release line main, not master
  • 810bbd2: ci: pin the functional-test server to a release whose watch API works
  • d145647: fix: disable SUBNET connectivity and licensing upsell paths
  • 5061c4f: rebrand: adopt Silo identity in CLI help and examples
  • c7f7706: docs: align governance files and package metadata with the fork
  • 65c71b2: test: default functional tests to a local server and add brand gate
  • c62a64d: fix: credit both MinIO and PGSTY in copyright notices
  • d205f88: docs: adopt no-CLA plus DCO contribution policy
  • 95326ce: docs: add related-projects table and polish contribution wording
  • d2c0db7: fix: remove the vendor encryption key and close the proxy-set path
  • 0c6704d: fix: repair a link and help text damaged by the brand sweep

2.17 - Silo 20260806 Released

The first release published under the Silo name: the completed rebrand with compatibility preserved, native health checking, a distroless image pilot, bundled mcli 20260806, full license compliance, and a provenance-gated release pipeline.

Version: RELEASE.2026-08-06T00-00-00Z · Commit: 3be10fcc1a44f6620ded0bd303461f9d688cca23

SILO 20260806 is the first release published under the Silo name. The previous release, 20260804, was the last one delivered as pgsty/minio; this release completes the cutover to github.com/pgsty/silo and renames every delivery surface — binary, packages, container images, systemd unit, Helm chart — while deliberately preserving every wire and configuration surface a MinIO deployment depends on. On top of the rename it adds native health checking (silo healthcheck), a single-binary distroless container image pilot, complete license-compliance materials in every artifact, and a release pipeline gated on compatibility snapshots and build provenance.

The release covers 28 commits after RELEASE.2026-08-04T00-00-00Z, changing 396 files with 27,188 insertions and 19,561 deletions. It passed a six-phase pre-release acceptance, including a real four-node TLS cluster migration from MinIO to Silo — with byte-verified data integrity, maintenance-gated rolling restarts, fault injection, and a full rollback rehearsal.

Highlights

  • The rebrand is complete, and compatibility is the contract. Repository, binary (/usr/bin/silo), packages (silo rpm/deb/apk), images (docker.io/pgsty/silo), and service (silo.service) are renamed; the S3 and admin APIs, /minio/* routes, MINIO_* environment variables, x-minio-* headers, on-disk .minio.sys format, and Go module paths are all preserved and frozen by a CI compatibility guard.
  • Native health checking: silo healthcheck [live|ready|cluster|cluster-read] probes the server’s own health API with correct exit codes, decoded quorum diagnostics, TLS auto-detection, and a --maintenance pre-drain gate — no shell, curl, or mc required in the container.
  • Distroless image pilot: pgsty/silo:<RELEASE>-distroless ships exactly one program — the silo binary — on gcr.io/distroless/static, with an exec-form HEALTHCHECK baked in and /data created writable in the image layer.
  • The classic image does not change behavior: same entrypoint, same bundled tools, mc ready local keeps working, and no HEALTHCHECK was added to it. It now bundles mcli 20260806.
  • Compliance completed: LICENSE and NOTICE ship in every package and image, CREDITS is regenerated from the actually-linked module set (291 modules) and guarded in CI, and the project adopts a no-CLA, DCO-based contribution policy.
  • Components refreshed: embedded SILO Console 2.1.1, silo-pkg 3.11.0, mcli 20260806, Go 1.26.5.
  • Provenance-gated releases: container images are built only from published, checksum- and attestation-verified release archives; image SBOMs and provenance attestations now cover the distroless variant too.

The rename

What changed, and what deliberately did not:

Renamed (delivery surface) Preserved (compatibility surface)
Repository: github.com/pgsty/silo (main branch) S3 API, admin API, and request signing behavior
Binary: /usr/bin/silo /minio/* routes, including /minio/health/* and metrics
Packages: silo-*.rpm, silo_*.deb, silo_*.apk MINIO_* environment variables and x-minio-* headers
Images: docker.io/pgsty/silo (+ -distroless) On-disk format (.minio.sys), erasure coding, versioning
Unit: silo.service (conflicts with, and supersedes, minio.service) Go module and import paths (github.com/minio/...)
Default config dir: ~/.silo (falls back to an existing ~/.minio) mc compatibility alias for the bundled mcli

The server presents its own identity — silo --version reports the AGPL-3.0 license, MinIO’s 2015-2025 copyright, PGSTY’s modification copyright, and the “based on MinIO technology” attribution — and every inherited connection to MinIO-operated services (the update feed and its signing key, SUBNET, telemetry) is severed rather than redirected. The container entrypoint translates the legacy minio argv token, so docker run pgsty/silo minio server /data keeps working.

A snapshot-based rebrand guard runs in CI: it fails on any drift, in either direction, across 334 route literals, 437 environment tokens, 84 headers, and 9,014 exported symbols.

Native health checking

The server binary can now probe its own health endpoints, which makes container health checks possible without any second binary — and is what the distroless image relies on:

silo healthcheck [FLAGS] [live|ready|cluster|cluster-read]
  • The check vocabulary maps 1:1 onto /minio/health/<path>; live (the default) answers “is this process serving,” ready adds KMS/etcd reachability when configured, and the cluster pair evaluates write/read quorum across every erasure set.
  • Exit codes are 0 (healthy) and 1 (anything else) — never the Docker-reserved 2. One diagnostic line decodes the server’s x-minio-server-status and quorum headers for docker inspect; --json emits a machine-readable verdict.
  • The probe target is derived the way the server derives its own listen address: --address / MINIO_ADDRESS, with HTTPS auto-detected from public.crt + private.key in the certs directory, or overridden wholesale with --url / MINIO_HEALTHCHECK_URL. The environment form exists because a probe process cannot see the server’s command line — if the server’s address or TLS comes from CLI arguments, one environment variable redirects the baked-in probe.
  • silo healthcheck --maintenance cluster answers the pre-drain question: exit 0 means the node can be taken down without losing HA; HTTP 412 (exit 1) means it cannot.
  • Certificate verification is skipped, matching the kubelet’s documented behavior for HTTPS probes, and the transport ignores HTTP_PROXY so loopback probes never route through a proxy.

Kubernetes needs none of this — kubelet httpGet probes hit /minio/health/live and /minio/health/ready from outside the container — and the cluster checks should stay out of per-container probes: they reflect cluster-wide quorum, not one process. The full design rationale, including verified endpoint semantics, is recorded in the health-check design note.

Distroless image pilot

Alongside the classic image, this release publishes a distroless variant: pgsty/silo:RELEASE.2026-08-06T00-00-00Z-distroless, plus a rolling distroless tag.

  • Base is gcr.io/distroless/static-debian12: CA certificates, tzdata, /tmp, and an /etc/passwd with a nonroot (65532) entry — no shell, no package manager, no libc. On top of it, exactly one program: /usr/bin/silo (plus the license set under /licenses/). The image is 128 MB versus the classic 199 MB.
  • The binary is the ENTRYPOINT; an exec-form HEALTHCHECK running silo healthcheck ready is baked in (interval 30s, timeout 10s, start-period 2m, retries 3), so Compose users get working depends_on: condition: service_healthy with zero configuration.
  • /data is created in the image layer, world-writable — there is no entrypoint left to repair volume ownership at runtime, and this is what makes every privilege mode work, --user included. This fixes, for the distroless variant, the non-root failure documented in #55.
  • Not supported in this variant: the deprecated MINIO_USERNAME/MINIO_GROUPNAME privilege-drop path (use --user or Kubernetes runAsUser), docker exec <c> sh debugging (use ephemeral-container tooling), and in-image mc (use the released mcli or the client image).
  • TLS: mount certificates at /tmp/.silo/certs (the container’s default certs directory) and both the server and the baked-in probe derive HTTPS from the same location; for CLI-configured servers, set MINIO_HEALTHCHECK_URL.

The classic image remains the default and is unchanged. If the pilot proves out, the distroless variant becomes the recommended image later; the decision record lives in the design note above.

Container images

The classic image was diffed field by field against pgsty/minio:RELEASE.2026-08-04T00-00-00Z: entrypoint, exposed ports, volumes, working directory, user, and (absent) health-check configuration are identical. Exactly three differences exist, all deliberate: Cmd is ["silo"] instead of ["minio"], the upstream update-verification key variable MINIO_UPDATE_MINISIGN_PUBKEY is removed (updates through upstream channels are permanently disabled), and HOME=/tmp is declared to match the entrypoint’s writable-home guarantee.

The bundled client is upgraded to mcli RELEASE.2026-08-06T00-00-00Z (with the mc alias preserved), pinned by per-architecture SHA-256 digests and verified against the published checksums at build time. Interoperability of the released mcli 20260806 against this server — multipart, versioning, presigned URLs, metadata/tags, user and policy administration — was verified as part of release acceptance.

Helm chart

The chart ships as silo 7.0.1, preserving rendered resource identity with the legacy chart across a simulated upgrade (verified by the migration guard over 7 rendered resources). Its default image tag now points at this release — docker.io/pgsty/silo is a fresh repository, so the inherited default could never have pulled. The chart still ships no liveness/readiness/startup probes; adding them is planned, and documented, in the design note’s follow-up phase.

Packaging and migration

RPM, DEB, and APK packages install exactly six files: /usr/bin/silo, silo.service, a sysusers definition (creating the silo system user), /etc/default/silo (marked config/noreplace), LICENSE, and NOTICE. RPMs are GPG-signed with the PGSTY maintainer key (9592A7BC 7A682E73 33376E09 E7935D8D B9BD8B20). RPM and DEB now carry a unified, PGDG-style 1PGSTY release segment — silo-<version>-1PGSTY.<arch>.rpm and silo_<version>-1PGSTY_<arch>.deb — replacing the inherited bare -1 on RPM and the missing revision on DEB; APK names stay bare because Alpine pkgrel admits only -r<integer>.

silo.service is designed for takeover: Type=notify (readiness is signaled by the server itself), Conflicts=minio.service + After=minio.service (starting Silo stops a running MinIO unit), and two environment files — /etc/default/minio is read first and /etc/default/silo overrides it — so an existing MinIO configuration is inherited without editing. For existing deployments whose data is owned by the minio user, the documented drop-in keeps ownership untouched:

# /etc/systemd/system/silo.service.d/10-legacy-user.conf
[Service]
User=minio
Group=minio
Warning

Distributed migrations must switch all nodes together.

Cluster bootstrap verifies that every node runs the same binary (by checksum). A mixed cluster — some nodes on Silo, some still on MinIO — does not form: the new node stays in activating, logging Expected Silo binary checksum ... seen: ... and Waiting for at least 1 remote servers with valid configuration, indefinitely. Stop MinIO on all nodes, then start Silo on all nodes (near-simultaneously). Once every node runs Silo, rolling restarts work normally — gate each one with silo healthcheck --maintenance cluster.

Migration troubleshooting, from the acceptance run: if Silo starts as the packaged silo user against a deployment whose TLS certificates live under the minio user’s home, it fails with HTTPS specified in endpoints, but no TLS certificate is found and restart-loops until the systemd start limit — the legacy-user drop-in above is the fix. Keep the MinIO package and unit installed (disabled) during the migration window: the rollback path — stop Silo, start MinIO — was rehearsed and reads all data written during the Silo window, because the migration touches neither data ownership nor format.

Components and dependencies

  • SILO Console 2.1.1 — the embedded console, selected from pgsty/silo-console while preserving the github.com/minio/console import path.
  • silo-pkg 3.11.0 — retains the policy/LDAP/certificate fixes including the LDAP-over-TLS repair tracked in #15.
  • mcli 20260806 — bundled in the image and released separately; see its release notes.
  • Go 1.26.5 — toolchain unchanged from 20260804.

Build, CI, and release pipeline

  • Compatibility as a CI gate: the rebrand guard snapshots routes, environment tokens, headers, metrics, storage/policy identifiers, and exported symbols, and fails on any unreviewed drift; companion scripts assert the delivery surface (binary path, unit contents, image layout) and that no live upstream endpoint remains in runtime code.
  • Release-image gate: every release-pipeline run builds both container images and asserts, among others: the distroless HEALTHCHECK survives into the image config (it is a Docker extension outside the OCI spec), /data ships world-writable, no shell and no /usr/bin/minio exist, Docker’s health state turns healthy from the baked probe alone, and SIGTERM still stops the server gracefully as root and as --user 1001:1001.
  • Provenance chain: images are built from the published release archives after checksum verification and gh attestation verify against the exact tag; per-architecture SBOMs and provenance attestations are pushed for the classic and distroless images; the distroless health-check gate runs before the multi-arch manifests are promoted.
  • Workflow runtime moved to Node 24 across CI actions.

Compatibility and upgrade notes

  1. Package upgrades are a takeover, not an in-place update. Install silo, keep /etc/default/minio as is (it is inherited), enable silo.service; starting it stops minio.service via the conflict relation. Data is untouched.
  2. Keep data ownership stable with the legacy-user drop-in above; do not chown storage or move certificates during migration.
  3. Distributed clusters: full-stop switchover only. See the warning above — mixed Silo/MinIO nodes do not form a cluster.
  4. Container users: the image is now docker.io/pgsty/silo; docker.io/pgsty/minio stays frozen at 20260804 as an archive. The classic image’s behavior is unchanged — including mc ready local health checks — and the distroless variant is strictly opt-in.
  5. Distroless differences are deliberate: no shell, no in-image mc, no MINIO_USERNAME path; health is native; servers configured via CLI arguments need MINIO_HEALTHCHECK_URL for the baked-in probe.
  6. Helm users: chart 7.0.1’s defaults now pull this release; override image.tag explicitly if you pin versions.
  7. Known and unchanged: the classic image still does not create /data in the layer, so fully non-root docker run against a Docker-managed volume fails as before (#55, fixed in the distroless variant); the inherited Postgres/MySQL legacy notification-migration limitation from the 20260804 notes still applies (#53).
  8. Pair with mcli 20260806 for the client side; older clients continue to work over the unchanged wire protocol.

Verification

This release was verified in stages, each with recorded evidence:

  • unit and end-to-end matrices for the health-check command: target derivation and precedence (flag/env/derived), real-TLS auto-detection, exit-code contract, JSON schema, timeout bounds, usage errors;
  • cluster-semantics verification on a four-node cluster: with 2 of 4 nodes stopped, cluster reports 503 with write-quorum=5 while cluster-read and live stay 200 — the write/read quorum split observed live, matching the erasure math;
  • image acceptance: the classic image diffed field-by-field against the 20260804 baseline; the distroless image asserted down to file inventory, exact health-check configuration, and root/non-root/TLS/env-override runtime scenarios;
  • an adversarial model-based code review of the new code, with every confirmed finding fixed and re-verified;
  • a six-phase pre-release acceptance concluding in a real migration: a Pigsty-deployed four-node TLS MinIO 20260804 cluster (16 drives, EC:4) was migrated to Silo via the packaged takeover path — reference data (multipart, versioned, tagged objects) read back byte-identical, four maintenance-gated rolling restarts, kill -9 fault injection with the load balancer serving 23/24 continuous IO rounds (the only failure in the kill second), Prometheus metrics continuity, and a full rollback to MinIO and back, proving the migration reversible.

Validation boundaries

Not proven by this release and not to be inferred: external LDAP/OIDC/KMS/etcd services (the only case where ready diverges from live was not exercised against a live KMS); amd64 packages were cross-built and payload-checked but not installed on a physical x86-64 host; the renamed Docker publish workflow (including the new SBOM/attestation lanes) has its first production run at this release’s publication; Windows and Intel macOS were not tested.

Artifacts

  • GitHub release RELEASE.2026-08-06T00-00-00Z at pgsty/silo, with checksummed platform archives, provenance attestations, and RPM/DEB/APK packages (GPG-signed RPMs);
  • docker.io/pgsty/silo:RELEASE.2026-08-06T00-00-00Z and latest; docker.io/pgsty/silo:RELEASE.2026-08-06T00-00-00Z-distroless and distroless — published on demand from the finished release;
  • companion releases: mcli 20260806, silo-pkg 3.11.0, embedded SILO Console 2.1.1;
  • design record: Native Health Checks and the Distroless Image.

Selected changes

  • 15def34dc, 77bdc4c0c: drop upstream delivery residue; present Silo identity and close inherited upstream services
  • 15ab10833: rename the delivery artifacts to silo and complete the package payload
  • 30749911b: ship the silo binary in the image and translate the legacy argv command
  • e071bb77e: replace the minio chart with a silo chart that preserves identity
  • bd8df5166: gate the rebrand on compatibility, packaging, and provenance evidence
  • 6613c2a3c: pin the external test fixtures and run the suites against the silo binary
  • fd2ca1c6d, c46b16ec6, c47733abc, f1c77d5a2: cut over to pgsty/silo and main; document the archived branch
  • 6740e6978: move the workflow actions onto the Node 24 runtime
  • b57275be3: adopt the no-CLA plus DCO policy and fix copyright terms
  • 62717d7bf, a6d6d9b02: update the embedded Console to 2.1.0, then 2.1.1
  • 6bd9cf77e: regenerate CREDITS from the linked module set and guard it in CI
  • 219670d31: ship LICENSE and NOTICE in every package and image
  • 2ff594f4b: add the native silo healthcheck subcommand
  • 4c34d2309: add the distroless image variant as a pilot
  • b6d47b739, 9462cce16: harden both per adversarial review; lint cleanup
  • 16b78eb4e: bundle mcli 20260806 and point the Helm defaults at this release
  • 062a91bee: pin the CREDITS module closure to the shipped linux target
  • 467931455: unify the rpm and deb release segment as 1PGSTY
  • b14ea22aa: match checksum manifest entries exactly in the image publish lane
  • 3be10fcc1: add a manual finalize lane refreshing SBOMs and checksums for signed Draft packages

Acknowledgments

Four contributors have code merged into this fork, and the Git history carries their authorship: @ZouhairCharef patched CVE-2026-34986 in go-jose (#18), @mfredenhagen patched CVE-2026-39883 in OpenTelemetry (#19), @pinginfo implemented Flush on trackingResponseWriter to repair bucket notification streaming (#34), and @waterkip repointed the documentation links to the Silo portal (#41).

A first release under a new name is also the right moment to thank everyone who has filed issues against this fork — bug reports, compatibility findings, and proposals alike, resolved and still open:

@mosesdd (#1), @Xavier-777 (#2, #17), @jiadzh (#3), @TLINDEN (#4), @AntonOfTheWoods (#5), @zylpsrs (#6), @nsanitate (#7), @makinikm (#9), @magicxor (#10), @spaceg00se-r (#11, #14), @heroes1412 (#13), @vampywiz17 (#15), @davinkevin (#20), @chalukyaj (#30), @cbornet (#31, #32), @jvasile (#33), @Kesavaambati (#35), @redfoxfox (#38), @kuldeep-link11 (#39, #40), @meesudzu (#42), @pmezhuev (#43), and @kh0mka (#51).

Several of this release’s headline items trace directly back to those reports: the bundled-client guarantee to #4 and #9, the LDAP-over-TLS repair to #15, the completed package payload to #33, GPG-signed RPMs to #43, the migration guide to #42, and the distroless /data fix to #55.

Pull requests still in flight deserve a mention too. @davinkevin’s distroless image PR (#21) anticipated this release’s pilot months in advance — the shipped variant supersedes that PR with the native health check built in, but the direction was proposed there first. Conformance PRs from @magicxor (#12) and @ycjlin (#37) are queued for review immediately after this release.

Everyone who has contributed to this fork is recorded in CONTRIBUTORS.md, which is now the project’s attribution record — GitHub generates no contributor graph for forks.

2.18 - Silo 20260804 Released

Internode storage containment, S3/IAM policy hardening, multipart correctness, streaming reliability, notification fixes, Go 1.26.5, and a rebuilt, signed release pipeline.

Version: RELEASE.2026-08-04T00-00-00Z · Commit: d88f46ccee345a9c2fabe2d221d9a9e56bc11aec

SILO 20260804 is a security, correctness, and release-engineering update to the pgsty/minio community fork. It completes the internode storage-containment work begun with CVE-2026-42600, prevents request-controlled values from impersonating server-calculated S3/IAM policy conditions, restores streaming flush behavior, fixes several multipart and versioning edge cases, hardens notification configuration migration, moves the build baseline to Go 1.26.5, and connects the server to the SILO-maintained Console, shared package, and mcli releases. The release pipeline was rebuilt to produce reproducible binaries and GPG-signed packages.

The release covers 50 commits after the pre-2026-06-18 baseline, changing 155 files with 9,241 insertions and 981 deletions. Every change was reviewed against the tagged commit and verified on macOS ARM64 and Linux AMD64, with GitHub CI green on the released HEAD.

Highlights

  • Internode containment completed: validates storage-REST message bodies, storage Grid frames, and peer-S3 Grid requests at the storage boundary, closing the remaining path, volume, erasure-metadata, panic, and unbounded-allocation defects left after removing ReadMultiple.
  • S3/IAM decisions now use effective values: client input can no longer shadow internal condition values; request tags and existing-object tags are separated; s3:signatureAge is confined to verified presigned requests; and s3:versionid follows the version the server actually acts on.
  • Bucket and object resources are separated: twelve sensitive bucket-level writes are no longer authorized through an object-only bucket/* resource pattern. A documented compatibility switch is available for migration.
  • Multipart compatibility and correctness improved: full-object checksum completion works without per-part checksums when the protocol permits it, zero-length multipart checksums are preserved, and duplicate part numbers are rejected instead of assembling duplicated data.
  • Streaming reliability restored: trackingResponseWriter now implements Flush correctly and records implicit HTTP 200 responses, repairing mcli watch, bucket-notification listeners, and S3 Select keep-alives affected by the inherited regression documented in the 20260618 release.
  • Notification configuration hardened: NATS and AMQP keys used by parsers and legacy migration are registered and round-trip correctly; libpq connection parameters are quoted safely; invalid-key errors no longer echo secret values.
  • Reproducible, signed release pipeline: binaries no longer embed the build machine’s paths, packages install under the canonical systemd path, and RPMs are GPG-signed. The container entrypoint now shuts down gracefully on every privilege path.
  • Release baseline refreshed: Go 1.26.5, klauspost/compress 1.18.7, Apache Thrift 0.24.0, SILO Console 2.0.0, silo-pkg 3.11.0, and mcli 20260804.

Security Hardening

Internode storage and Grid containment — SN-2026-002

Removing the obsolete ReadMultiple endpoint in 20260618 closed one reachable path but did not close the underlying defect class. Storage-REST request bodies and Grid RPC frames do not pass through the HTTP query-validation middleware, and peer-S3 RPCs can bypass the storage-REST wrapper entirely.

This release moves containment to the storage boundary and validates every caller-controlled path, volume, erasure parameter, part size, shard length, and allocation length before use. The fixes include:

  • reject traversal on both path and volume axes, including Windows volume-root aliases;
  • cover peer-S3 bucket RPCs that reach drives without the storage-REST wrapper;
  • reject zero or unusable data/parity/block-size combinations before shard arithmetic;
  • reject negative part sizes and truncated shards instead of reporting them healthy;
  • cap storage-REST ReadFile allocations at 5 GiB;
  • bound other allocations derived from internode declarations;
  • contain panics in deadline-bounded storage work without blocking the caller;
  • preserve ReadParts errors across keep-alive responses and avoid the empty-part trace panic.

These routes require cluster-root or internode credentials and are registered only in distributed-erasure deployments. Single-node S3 behavior is unchanged. See Internode Path Containment Audit for the protocol-surface analysis.

Effective policy-condition values — SN-2026-003

The policy condition map historically mixed values calculated by the server with raw request entries. A client-controlled spelling could therefore shadow or synthesize an internal condition value. SILO 20260804 pairs silo-pkg 3.11.0’s exact-key lookup rule with server-side source normalization:

  • internal condition names cannot be supplied as arbitrary client values;
  • s3:prefix, s3:delimiter, and s3:max-keys come from their effective query inputs;
  • header-backed x-amz-* conditions do not accept unrelated query substitutes;
  • when storage class or upload tagging supports both forms, an explicitly present Header wins, including an empty Header;
  • s3:ExistingObjectTag/* comes only from stored object metadata;
  • s3:RequestObjectTag/* is bound to the tag input consumed by the relevant operation;
  • s3:signatureAge is exposed only after verified SigV4 presigned authentication calculates it;
  • s3:versionid is absent when no version is named and is rebound per DeleteObjects entry to the effective resolved version.

The version-ID behavior closes the fail-open trap that a superficial “omit empty values” fix would have created for Multi-Delete. See Absent Is Not Empty.

Bucket/object resource boundary — SN-2026-004

The IAM matcher used to append a slash to a bucket-level request, allowing an object-only resource such as arn:aws:s3:::bucket/* to authorize selected bucket-level operations. This release withholds twelve sensitive writes from that pattern on Allow statements:

PutBucketPolicy, DeleteBucketPolicy, PutBucketObjectLockConfiguration, PutBucketVersioning, PutReplicationConfiguration, PutBucketLifecycle, DeleteBucket, ForceDeleteBucket, PutBucketCors, DeleteBucketCors, PutBucketQOS, and PutInventoryConfiguration.

Deny and NotResource behavior is unchanged. Read/list operations, CreateBucket, bucket tagging, default encryption, and notification configuration remain compatible. Built-in policies use Resource: "*" and are not affected.

Warning

Policy migration required for custom bucket grants

If a custom policy grants one of the twelve actions — often through s3:* — using only arn:aws:s3:::bucket/*, add the bare bucket ARN:

"Resource": ["arn:aws:s3:::bucket", "arn:aws:s3:::bucket/*"]

MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on restores the historical matcher while policies are migrated. It also restores the historical over-grant, so use it only as a temporary rollback control.

Trusted client-address boundary

MINIO_API_TRUSTED_PROXIES provides an enforceable, opt-in boundary for aws:SourceIp, audit remotehost, event notification Host, and the client address shown by mcli admin trace:

  • set it to an address/CIDR list to trust forwarding headers only from those peers and walk forwarding chains from right to left;
  • set it to none to ignore all forwarding headers;
  • leave it unset to preserve historical behavior exactly.

The old _MINIO_API_XFF_HEADER=off switch still suppresses only X-Forwarded-For; it does not protect against X-Real-IP or RFC 7239 Forwarded. If IP-based policy is part of your security boundary, configure trusted proxies explicitly and prevent direct access to the S3 API port. Multi-node deployments should allow their own node addresses. See Client Source Address Trust.

S3 and Storage Correctness

Multipart upload

  • CompleteMultipartUpload accepts the S3 full-object checksum mode when the completed request supplies no per-part checksums and the upload metadata does not require them.
  • The checksum of a zero-length multipart object is retained instead of being discarded as empty metadata.
  • Part numbers must be strictly increasing. Duplicate entries such as [1,1] now return InvalidPartOrder instead of consuming the upload and assembling the same part twice. Legal part lists with gaps or a non-1 start remain accepted. See Duplicate Part Numbers.

Object reads and buffer ownership

  • erasure reads again pool buffers only where ownership permits reuse;
  • update downloads return caller-owned buffers instead of exposing data that can be overwritten after return;
  • the old HTTP streaming helpers orphaned by ReadMultiple removal are deleted after reference and platform-tag checks.

HTTP response tracking and S3 Select

  • trackingResponseWriter.Flush() delegates to the underlying flusher and commits the response state correctly;
  • the first implicit write records HTTP 200, preserving audit and metric accuracy;
  • S3 Select tests no longer race a client parser against response-body ownership;
  • CSV, JSON, and Parquet selection paths remain covered, including range/error and keep-alive behavior.

The inherited silent-flush regression called out in SILO 20260618 is therefore fixed in this release.

IAM, Versioning, and Audit Behavior

  • DeleteObject and each entry in DeleteObjects evaluate s3:versionid against the effective version selected by the server.
  • Request tags can no longer impersonate existing-object tags during policy evaluation.
  • The merrs tag is restored when dangling-object deletion records are emitted, preserving the intended audit classification.
  • Bucket-policy and IAM paths share the hardened condition-source rules while retaining their established S3 routing and error behavior.

Notification Configuration

  • registers the NATS user_credentials, nkey_seed, and tls_handshake_first keys read by the parser;
  • separates the legacy NATS environment-variable spelling from the stored config key;
  • repairs NATS migration round trips and the AMQP immediate/internal mapping;
  • adds a mechanical audit that compares keys read and written by notification code with each subsystem’s registered schema;
  • quotes libpq connection-string parameters so whitespace, quotes, and backslashes retain their intended value;
  • prevents invalid-key diagnostics from echoing secret values.

See Notify Keyspace Registration.

Warning

Known legacy migration limitation

The inherited Postgres and MySQL legacy migration functions still write the unregistered host, port, username, password, and database fields. A migrated configuration can therefore fail validation on the next load, and notification target loading is fail-fast across subsystems. This predates 20260804, but upgrades from pre-connection-string database notification configurations must be reviewed and converted before restart. The stored password field may contain a plaintext database password.

Components and Dependencies

  • Go 1.26.5: includes security fixes in crypto/tls and os plus compiler, runtime, networking, and syscall corrections.
  • klauspost/compress 1.18.7: refreshes the compression stack used by object and archive paths.
  • Apache Thrift 0.24.0: updates the dependency compiled through Parquet support.
  • go-systemd 22.6.0: deliberately retained instead of 22.7.0 because the later version introduced a NetBSD clock dependency incompatible with the supported cross-build matrix.
  • SILO Console 2.0.0: the embedded console is selected from pgsty/silo-console while preserving the compatible github.com/minio/console import path.
  • silo-pkg 3.11.0: provides the companion policy, LDAP, certificate, RNG, and time-format fixes while preserving the github.com/minio/pkg/v3 module path.
  • mcli 20260804: the embedded client comes from pgsty/mc; release images expose it as mcli and keep the mc compatibility alias.

See the companion release notes for silo-pkg 3.11.0, mcli 20260804, and SILO Console 2.0.0.

Build, CI, and Packaging

This release rebuilt the release pipeline for reproducibility and supply-chain integrity:

  • Graceful container shutdown on every path. The entrypoint’s custom UID/GID branches now exec into the server so it runs as PID 1 and receives SIGTERM directly; previously those branches left an intermediate shell as PID 1 and the server was killed at the container stop timeout. A CI smoke test builds the release runtime image and asserts graceful shutdown on both the default and drop-privilege paths.
  • Reproducible binaries. Release binaries no longer embed the build machine’s GOPATH/GOROOT, so -trimpath holds and a third party rebuilding the tag gets matching bytes. The published Linux binary contains no build-host path.
  • Hardened release workflow. The release tag is passed through the environment and whitelisted rather than spliced into the shell, the build is checked out at the tag being released, and an untracked shadow GoReleaser config that could publish or move latest out of band was removed.
  • Honest gates. CI gates build, vet, unit tests, lint, generation drift, race tests, and cross-compilation; the cross-compile matrix is aligned to the exact set of published targets; and the lint and dependency-install steps now fail on real errors instead of masking them.
  • Signed, canonical packages. RPM, DEB, and APK packages are produced with nFPM under the PGSTY identity, the systemd unit installs at /usr/lib/systemd/system/minio.service with Type=notify, and RPMs are GPG-signed offline with the PGSTY maintainer key (fingerprint 9592A7BC 7A682E73 33376E09 E7935D8D B9BD8B20).
  • Release and container publication remain separate gates. GoReleaser produces the platform archives, checksums, and packages; the multi-architecture image is published on demand from the finished release. A local snapshot does not prove a public release or image exists.

Compatibility and Upgrade Notes

  1. Keep every node on one release during a cluster rollout. Internode validation changed across storage-REST and Grid surfaces; mixed binaries were not production-tested.
  2. Audit custom IAM policies. Add the bare bucket ARN for the twelve protected bucket writes. Use MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on only as a temporary migration control.
  3. Configure client-address trust deliberately. If aws:SourceIp or audit attribution matters, set MINIO_API_TRUSTED_PROXIES and close direct network paths around the proxy.
  4. Review legacy database notification settings. Convert Postgres/MySQL host/user/password fields to the supported connection-string format before restart.
  5. Expect duplicate multipart completion entries to fail. Clients sending the same part number more than once now receive InvalidPartOrder instead of a corrupted successful object.
  6. Use the matching mcli. The 20260804 client disables self-update and must be upgraded through packages or GitHub Releases; mcli update remains as a compatibility command but exits non-zero.
  7. RPM users can enable signature verification. Packages are signed with the maintainer key above; import it before enabling gpgcheck for the SILO packages.

Verification

Changes were reviewed against the tagged commit and re-verified rather than trusted from prior reports:

  • git diff --check, gofmt, module verification, and YAML/shell syntax;
  • go build ./..., go vet ./..., project lint, and govulncheck ./...;
  • full go test ./..., the complete race suite, and repeated race tests over storage, policy, notification, HTTP tracking, and S3 Select changes;
  • generator idempotence plus deliberate stale-source and untracked-output counterexamples;
  • cross-compilation across every published target;
  • Linux AMD64 native full tests, targeted race tests, live S3/mcli smoke tests (create/upload/download/copy, range, versioning, delete markers, health checks, graceful shutdown, restart persistence), and systemd notify behavior;
  • release-artifact verification: GitHub CI green on the released HEAD, reproducible binaries with no build-host path, the systemd unit installed at /usr/lib/systemd/system, and RPM signatures validated with rpmkeys --checksig.

govulncheck found no vulnerability reachable from the Server or mcli code. One module-level notice remains for the unmaintained golang.org/x/crypto/openpgp package (GO-2026-5932); that package is not imported into these binaries.

Validation boundaries

The following were not proven by this release and must not be inferred from cross-compilation or unit tests:

  • native Windows execution and Windows filesystem semantics;
  • Intel macOS and physical Linux ARM64 hosts;
  • a production multi-node rolling upgrade, site replication, or lifecycle expiration run;
  • real reverse-proxy chains and direct-ingress isolation;
  • external LDAP, OIDC, KMS, STS, Postgres, MySQL, NATS, and AMQP services;
  • installing and upgrading the signed package under a real systemd host.

Artifacts

  • GitHub release RELEASE.2026-08-04T00-00-00Z with checksummed platform archives for Linux, Darwin, and Windows on amd64 and arm64;
  • RPM, DEB, and APK packages under the PGSTY identity, with GPG-signed RPMs;
  • docker.io/pgsty/minio:RELEASE.2026-08-04T00-00-00Z and the release-selected latest tag, published on demand from the release;
  • matching SILO Console 2.0.0, silo-pkg 3.11.0, and mcli 20260804 references.

Selected Changes

  • ca7baa670, 80e8eaa42, b6f70ab08: validate internode paths, erasure metadata, and allocation sizes
  • a36fd8fff: contain panics in deadline-bounded storage work
  • 2f55347f7: bind S3/IAM policy conditions to effective request values
  • 744a9dcd7: bind s3:versionid to the effective object version
  • 97b7d2804: enforce the bucket/object resource boundary
  • fe6dc4780: add the trusted-proxy client-address boundary
  • 22c1e41fd: reject duplicate multipart part numbers
  • c8590413f, 3e14733f1: restore full-object and zero-length multipart checksum behavior
  • 8069a32ac, 65795ee1f: restore response commit and streaming flush semantics
  • 162ded343, 0c14d8151: repair notification key registration and libpq quoting
  • 924717926, 89d346bf5: restore safe buffer pooling and returned-buffer ownership
  • 3b8a55dee: exec into the dropped-privilege process so signals reach the server
  • 2ca4971d9: stop stamping the build machine’s paths into the binary
  • 4c185d5a6, e064b5555: harden the release workflow and remove the shadow config
  • aa5139369: install the systemd unit under /usr/lib
  • 11d79fddc, ca674a696, 021110b45, d88f46cce: gate build, vet, tests, lint, generation, race, and cross-compilation, and smoke-test the release image

2.19 - mcli 20260804 Released

Self-update disabled, SUBNET debug-log credential leak fixed, containers built from fork source, packaging migrated to nFPM, and signed RPMs.

Published: 2026-08-04 · Version: RELEASE.2026-08-04T00-00-00Z

This is the first release of the pgsty/mc community fork since 20260417. It fixes a credential leak in debug logging, severs every remaining connection between the client and upstream release channels, moves containers and packages onto artifacts this fork builds itself, and migrates packaging from MinIO’s pkger to standard nFPM — with GPG-signed RPMs for the first time.

Upstream minio/mc was archived in July 2026. Its final commit, 77f82e18, is exactly this fork’s base, and upstream never cut a release containing it — so this build is strictly newer than any official mc binary ever published.

Warning

Behavior change

mcli update self-update is disabled in this fork. The command remains for script compatibility and still accepts its original arguments, but it no longer contacts the network or replaces its own binary; it prints an explicit notice and always exits with status 1. Upstream mc update exited 0 when already up to date, so drop the call from any script that treats a non-zero exit as failure. Upgrade through the Pigsty package repository or GitHub Releases.

The automatic version check that ran against upstream release feeds on every invocation has also been removed entirely. The MC_UPDATE and MINIO_UPDATE environment variables are no longer consulted.

Major Changes

  • Self-update disabled, upstream release channels severed: the minio/selfupdate and aead.dev/minisign dependencies and all binary-replacement logic are gone, along with the update notifier and the FIPS/non-FIPS update paths. The update command survives as a compatibility shell, and the runtime helpers (Docker / DCOS / Kubernetes / source-build detection) moved to a dedicated cmd/runtime-info.go. The client previously reached out to upstream release feeds on every invocation to print an upgrade hint; there is now no outbound release probing at all.
  • Containers and artifacts fully localized: the default image is built from the checked-out fork source, and hotfix binaries are copied from the local build context — no upstream prebuilt binaries are downloaded. The upstream publishing files Dockerfile.release, Dockerfile.release.old_cpu, and docker-buildx.sh were removed, and the obsolete MinIO hotfix upload target is disabled.
  • Packaging migrated to nFPM: replaced MinIO’s pkger with standard nFPM. Artifact layout and install path are unchanged (/usr/local/bin/mcli, package name mcli, YYYYMMDDHHMMSS.0.0 version scheme), but the vendor is now PGSTY, the license uses the SPDX identifier AGPL-3.0-or-later, and the Debian Section moved from empty to utils.
  • RPMs are now GPG-signed: RPMs are signed offline with the maintainer key (fingerprint 9592A7BC7A682E7333376E09E7935D8DB9BD8B20). All package metadata is asserted before signing, and the signature is re-verified with checksums regenerated afterwards. DEB and APK packages carry no package-level signature; their trust anchor lives at the repository layer.
  • Build provenance hardened: every previously published binary was stamped by the Go toolchain as built from a modified working tree (vcs.modified=true), which broke the link between an artifact and its Git tag. This release fixes that and adds an enforcing check to both the release and test pipelines, so every binary is traceable to an exact commit.

Security Fixes

  • SUBNET credentials redacted in debug logs: with --debug enabled, SUBNET HTTP exchanges are printed in full. Previously the api-key / api_key query parameters, authentication headers, and response bodies all reached the log in clear text — and SUBNET’s authentication and registration endpoints return API keys, licenses, and tokens in their responses. Both parameter spellings and duplicate values are now masked uniformly, sensitive response headers are redacted, and SUBNET response bodies are excluded from debug dumps entirely. The leak is inherited from upstream and present in every previous release, upstream mc included: if you have ever shared --debug output of SUBNET commands (mcli license ... / mcli support ...), treat the API keys and licenses in it as exposed and rotate them.
  • Redaction isolated from caller state: debug tracing now dumps copies of the request and response, so redaction cannot mutate objects the caller still holds. Zero-length, fixed-length, and unknown-length response bodies are all covered, and callers can still read the response normally.

Dependency Updates

This cycle’s dependency work is security maintenance, not routine hygiene: every bump below except the term / mod / sync / tools refresh closes at least one published advisory in the Go vulnerability database, and govulncheck reports zero known vulnerabilities reachable from this release’s code. No security advisory has ever been published for minio/mc, minio-go, madmin-go, or minio/pkg themselves.

  • Go build baseline upgraded from 1.26.2 to 1.26.5 (the newest 1.26.x at release time), picking up the 1.26.3–1.26.5 security batches — including GO-2026-4970 (symlink-based root escape in os) and GO-2026-5856 (Encrypted Client Hello privacy leak in crypto/tls), the two most relevant to an S3 client that writes local files and speaks TLS.
  • github.com/klauspost/compress from v1.18.5 to v1.18.7 (closes GO-2026-5841).
  • github.com/prometheus/prometheus from v0.310.0 to v0.311.3 (closes GO-2026-5264, GO-2026-5381, GO-2026-5710).
  • google.golang.org/grpc from v1.79.3 to v1.82.1 (closes GO-2026-6061), with the genproto family refreshed alongside.
  • The golang.org/x/* family refreshed across the board: crypto v0.49.0v0.53.0 (the 14-advisory GO-2026-5005…5033 batch), net v0.52.0v0.56.0 (GO-2026-5025…5030 and GO-2026-5942), sys v0.42.0v0.46.0 (GO-2026-5024), text v0.35.0v0.39.0 (GO-2026-5970), plus term, mod, sync, and tools.
  • Removed aead.dev/minisign and github.com/minio/selfupdate, and synchronized the third-party credits file.

Engineering and Delivery

  • Integration test dependencies pinned: CI no longer downloads the MinIO server from a mutable upstream URL. It now uses a versioned pgsty/minio release archive verified by its SHA-256 digest, with Go pinned to 1.26.5.
  • Release pipeline verification: a packaging validation workflow compares the binary inside all three package formats byte-for-byte against the build output, and checks package names, checksums, architecture fields, and every metadata field. The expected RPM metadata is sourced from the signing script itself, so configuration drift cannot strand a release part-way through signing.
  • CI supply-chain hardening: every GitHub Action is pinned to a commit SHA with dependabot keeping them current, workflow permissions are narrowed to read-only, and a stale workflow pointing at the upstream organization’s project board was removed.
  • Documentation: the English and Chinese READMEs now state this fork’s distribution channels and self-update policy explicitly, and installation instructions that would silently install upstream mc were removed.
Note

Known issue

mcli watch (bucket event notification) receives no events against any published pgsty/minio server release. The cause is a silent streaming-flush regression on the server side, inherited from upstream — it is not a client problem, and the previous mcli release is affected identically. The fix was merged to the server’s master on 2026-07-29 but has not shipped in a published server release. See the SILO 20260618 release notes and PR #34.

Separately, this fork inherits upstream’s unfixed defects, and with the upstream repository archived they can only ever be fixed here. The most serious is minio/mc#5139: mirror --remove --watch can delete a live object from the target when a non-current version of it is removed from the source. Exercise caution combining --remove --watch on versioned buckets.

  • 9603ee3: fix: redact SUBNET secrets in HTTP debug logs
  • f6ae2b0: fix: disable self-update in Pigsty builds
  • c05a6e4: build: update Go deps and toolchain to 1.26.5
  • 1f105aa: build: use local fork artifacts for containers
  • 1182da5: ci: pin fork integration test dependencies
  • 9ee207f: docs: clarify Pigsty fork distribution channels
  • 0686cd8: fix: isolate SUBNET debug redaction
  • ad10a2a: build: complete local Docker context isolation
  • 5f54221: docs: update mc README and cn version
  • 02c0305: build: migrate release packaging to nFPM
  • 4c4dcc4: build: harden release provenance and package metadata

2.20 - Silo 20260618 Released

LDAP STS hardening, complete S3 Select record limits, ReadMultiple removal, Go 1.26.4, and security dependency updates.

Published: 2026-06-18 · Version: RELEASE.2026-06-18T00-00-00Z

This release is a security and dependency-maintenance update for the pgsty/minio fork. It hardens LDAP STS throttling, completes S3 Select oversized-record enforcement, removes the obsolete ReadMultiple internode storage-REST API, upgrades the Go build baseline to 1.26.4, and refreshes Go module dependencies to pick up additional third-party security fixes.

Note

Note

Known issue: this release — like every earlier community release since RELEASE.2025-12-03T12-00-00Z — carries a silent streaming-flush regression inherited from upstream that breaks mc watch / bucket-notification listeners and S3 Select keep-alives. There is no workaround. The fix was merged to master on 2026-07-29 but has not shipped in a published server release; see PR #34 for the implementation.

Major Changes

  • Remove the obsolete ReadMultiple storage-REST API: the legacy /rmpl internode endpoint is removed rather than patched in place, including its route, handler, client wrapper, storage interfaces, xlStorage methods, generated datatypes, and related metric. No production caller is expected after upstream multipart handling moved to ReadParts, but clusters should still run a consistent release during rolling upgrades.
  • Complete S3 Select oversized-record enforcement: JSON Lines input now uses the bounded reader path, so oversized records are rejected consistently instead of bypassing limits on SIMD-capable CPUs. S3 Select stream errors now preserve the intended error code and wrap JSON parser failures as JSONParsingError.
  • Harden LDAP STS rate-limit source bucketing: throttling is now keyed only by source IP, avoiding username-shared buckets that could be drained by one client to lock out a legitimate user. Trusted-proxy handling now resolves X-Forwarded-For from right to left, rejects catch-all trusted-proxy CIDRs, ignores RFC 7239 Forwarded, and documents the X-Real-IP deployment contract.
  • Refresh the Go runtime and module baseline: release, hotfix, goreleaser, and old-CPU Docker builds now use golang:1.26.4-alpine; go.mod is updated to Go 1.26.4; and dependencies are refreshed across NATS, Prometheus, Azure SDK, Apache Thrift, gRPC, OpenTelemetry, Google API/auth, Go x/*, and related transitive libraries.

Direct Security Fixes

  • CVE-2026-42600: remove the obsolete ReadMultiple storage-REST API to close the legacy internode file-read path exposed through /rmpl.
  • CVE-2026-39414: complete oversized S3 Select record enforcement for JSON Lines inputs and preserve correct S3 Select error semantics.
  • CVE-2026-33419: further harden LDAP STS rate-limit accounting and trusted-proxy source-IP handling.

Dependency Security Updates

  • Update github.com/Azure/go-ntlmssp from v0.1.0 to v0.1.1, fixing CVE-2026-32952, where malformed NTLM challenges could panic a Go process.
  • Update github.com/apache/thrift from v0.22.0 to v0.23.0, fixing CVE-2026-41602 in the Go TFramedTransport implementation.
  • Update github.com/nats-io/nats-server/v2 from v2.11.1 to v2.11.15, absorbing the NATS 2.11.x security patch line. Notable fixes include pre-auth WebSocket and leafnode denial-of-service issues, MQTT authorization issues, JetStream management API authorization hardening, credential exposure fixes, and request identity-spoofing fixes, including CVE-2026-27889, CVE-2026-29785, CVE-2026-33217, CVE-2026-33218, CVE-2026-33222, and CVE-2026-33247.
  • Update github.com/prometheus/prometheus from v0.310.0 to v0.311.3, absorbing Prometheus security fixes for remote-read denial of service, stored XSS in UI surfaces, and remote-write configuration secret exposure, including CVE-2026-42154, CVE-2026-44903, CVE-2026-42151, and CVE-2026-40179.
  • Upgrade the release build baseline through Go 1.26.4 and refresh supporting Go module families, including golang.org/x/crypto, golang.org/x/net, golang.org/x/sys, golang.org/x/text, google.golang.org/grpc, and OpenTelemetry. These updates keep the fork aligned with patched upstream dependency baselines even where the previously pinned version was already past the specific public advisory range.
  • 5e40665: fix: harden LDAP STS rate-limit source bucketing
  • fd69c89: fix: complete CVE-2026-39414 S3 Select record limit enforcement
  • 73ac524: fix: CVE-2026-42600 remove ReadMultiple storage-REST API
  • df627ff: fix: bump Go toolchain to 1.26.4
  • 3e61b1d: chore: update Go module dependencies

2.21 - Silo 20260417 Released

Security hardening across OIDC, LDAP STS, S3 Select, replication metadata, unsigned-trailer flows, and the Go toolchain.

Published: 2026-04-17 · Version: RELEASE.2026-04-17T00-00-00Z

This release focuses on security hardening and compatibility tightening. It bundles fixes across OIDC, LDAP STS, S3 Select, replication metadata handling, unsigned-trailer flows, the Snowball upload path, and multiple dependency- and Go toolchain-related security issues, while also incorporating the LDAP TLS regression fix and a cleanup of community-fork documentation.

Major Changes

  • Tighten the identity-authentication flow: OIDC / WebIdentity now accepts only asymmetrically signed ID Token values backed by the IdP JWKS; symmetrically signed tokens such as HS256 are no longer accepted. LDAP STS also now hides the distinction between unknown-user and bad-password failures to reduce username-enumeration risk.
  • Update LDAP STS rate limiting: limits now apply to both source IP and normalized username, and successful requests no longer consume quota incorrectly. By default MinIO now uses only the socket peer address as the source and no longer trusts X-Forwarded-For, X-Real-IP, or Forwarded; to rate-limit by real client IP, configure MINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIES explicitly.
  • Make upload and write paths stricter: presigned query parameters can no longer be combined with unsigned-trailer PUT or multipart uploads. Snowball auto-extract now also performs full signature validation on the unsigned-trailer path and rejects anonymous or forged-signature requests.
  • Prevent replication metadata spoofing: internal X-Minio-Replication-* headers attached to ordinary PUT / COPY requests are now rejected or ignored, and only trusted replication flows may write the related internal metadata.
  • Clarify S3 Select error semantics: oversized CSV and line-delimited JSON records now return OverMaxRecordSize directly instead of the generic InternalError; clients or alerting rules that depend on the old error code should be adjusted.
  • Upgrade the runtime and dependency baseline: fix the regression where ldaps:// did not correctly apply TLS settings, replace minio/pkg/v3 with pgsty/minio-pkg/v3, and pin several critical dependencies that are prone to breaking changes. The release also upgrades go-jose, go.opentelemetry.io, and Go 1.26.2 to unify the build and release baseline.
  • Refresh documentation and security guidance: update SECURITY.md, VULNERABILITY_REPORT.md, docs/sts/ldap.md, and related documents, add a security advisory index, and switch upstream minio/minio references in the security guidance over to pgsty/minio.

Fixed CVEs

  • CVE-2026-34986: upgrade go-jose to v4.1.4 and fix known security issues in the JWT / JOSE dependency chain.
  • CVE-2026-39883: upgrade the go.opentelemetry.io dependency stack to fix the PATH-hijacking risk.
  • CVE-2026-33322: restore the strict JWKS-only OIDC JWT verification path to block keyring injection and algorithm-confusion risk.
  • CVE-2026-33419: systematically harden LDAP STS authentication, rate limiting, source-address identification, and accounting logic across four follow-up fixes.
  • CVE-2026-34204: reject injection of X-Minio-Replication-* metadata by untrusted requests to prevent objects from being written with invalid replication state.
  • CVE-2026-39414: reject oversized S3 Select records early to avoid continued buffering and parsing of abnormal inputs.
  • GHSA-hv4r-mvr4-25vw: close the unsigned-trailer query-auth bypass.
  • GHSA-9c4q-hq6p-c237: harden unsigned-trailer authentication and signature validation in Snowball auto-extract scenarios.
  • CVE-2026-32280, CVE-2026-32281, and CVE-2026-32283: upgrade Go to 1.26.2 and absorb the upstream toolchain and stdlib security fixes.
  • ce1c537: fix: pin deps with breaking changes and fix LDAP TLS regression (#15)
  • 68e0ba9: fix: upgrade go-jose to v4.1.4 to patch CVE-2026-34986
  • 1869bd3: fix: CVE-2026-39883 upgrade go.opentelemetry.io
  • ff58df9: merge: PR #18 upgrade go-jose to v4.1.4 for CVE-2026-34986
  • e4fa063: merge: PR #19 upgrade go.opentelemetry.io for CVE-2026-39883
  • d24f449: fix: CVE-2026-33322 restore JWKS-only OIDC JWT verification
  • 3b950f8: fix: CVE-2026-33419 harden LDAP STS auth
  • 56fa63b: fix: CVE-2026-34204 reject untrusted replication metadata
  • 3252d5b: fix: CVE-2026-39414 reject oversized S3 Select records
  • f444b6f37: fix: fake CVE-2026-40027 block unsigned-trailer query auth bypass
  • efb6e5b00: fix: fake CVE-2026-40028 harden snowball unsigned-trailer auth
  • db4c0fd: fix: CVE-2026-32280/CVE-2026-32281/CVE-2026-32283 upgrade Go to 1.26.2
  • 18b712d: fix: CVE-2026-33419 preserve LDAP STS rate limits on success
  • 9e10f6d: fix: CVE-2026-33419 harden LDAP STS rate-limit source IP
  • f441108: fix: CVE-2026-33419 tighten LDAP STS rate-limit accounting
  • f48dbe7: docs: refresh security guidance and fork references

2.22 - Silo 20260325 Released

Packaging, stability, LDAP TLS, Docker image, and dependency-security maintenance.

Published: 2026-03-25 · Version: RELEASE.2026-03-25T00-00-00Z

This is a maintenance release centered on packaging, stability, and security disclosure. It improves the shipping artifacts, fixes an LDAP TLS regression, and explicitly documents the secure dependency set carried by the release.

Major Changes

  • Bundle mcli/mc into the Docker image and add checksum verification for a better out-of-the-box image experience.
  • Fix the LDAP TLS regression affecting ldaps:// deployments so TLS settings are correctly honored.
  • Remove inherited upstream CI/CD workflows that are no longer used in the community-maintained fork.
  • Pin several critical dependencies to avoid further fallout from upstream breaking changes.

Fixed CVEs

  • CVE-2026-24051: the release notes explicitly call out go.opentelemetry.io/otel/sdk v1.42.0, which avoids the macOS PATH-hijacking arbitrary code execution issue.
  • CVE-2025-10543: the release notes explicitly ship github.com/eclipse/paho.mqtt.golang v1.5.1, fixing incorrect MQTT packet encoding for oversized UTF-8 strings.
  • CVE-2025-58181: the release notes explicitly ship golang.org/x/crypto v0.49.0, fixing unbounded memory consumption in ssh GSSAPI authentication handling.
  • f2f9a40: add mcli/mc from pgsty/mc to Docker image
  • ce1c537: fix: pin deps with breaking changes and fix LDAP TLS regression (#15)
  • ee55e53: remove upstream CI/CD workflows inherited from minio/minio

2.23 - Silo 20260321 Released

Go 1.26.1, stricter compiler and linter compatibility, and a broad security dependency refresh.

Published: 2026-03-21 · Version: RELEASE.2026-03-21T00-00-00Z

This maintenance release is built around the Go 1.26.1 upgrade and a broad dependency refresh. Beyond stricter compiler and linter compatibility fixes, it also delivers the most substantial security dependency refresh in the current release line.

Major Changes

  • Upgrade the build environment from Go 1.26.0 to Go 1.26.1.
  • Refresh direct and indirect dependencies to converge on the newer toolchain.
  • Fix linter and test issues exposed by the stricter Go 1.26.1 checks.

Fixed CVEs

  • CVE-2026-27137: Go stdlib 1.26.0 -> 1.26.1 fixes incomplete email-constraint enforcement in crypto/x509.
  • CVE-2026-27138: Go stdlib 1.26.0 -> 1.26.1 fixes a crypto/x509 panic triggered by malformed certificates.
  • CVE-2026-25679: Go stdlib 1.26.0 -> 1.26.1 fixes insufficient validation of IPv6 host literals in net/url.
  • CVE-2026-27139: Go stdlib 1.26.0 -> 1.26.1 fixes FileInfo metadata escaping the Root boundary in os.
  • CVE-2026-27142: Go stdlib 1.26.0 -> 1.26.1 fixes missing URL escaping in html/template for meta refresh content.
  • CVE-2026-26958: filippo.io/edwards25519 v1.1.0 -> v1.2.0 fixes incorrect or undefined MultiScalarMult behavior.
  • CVE-2025-10543: github.com/eclipse/paho.mqtt.golang v1.5.0 -> v1.5.1 fixes incorrect MQTT packet encoding for oversized UTF-8 strings.
  • CVE-2026-24051: go.opentelemetry.io/otel/sdk v1.38.0 -> v1.42.0 fixes the macOS PATH-hijacking arbitrary code execution issue.
  • CVE-2026-33186: google.golang.org/grpc v1.77.0 -> v1.79.3 fixes authorization bypass caused by a missing leading slash in the HTTP/2 :path pseudo-header.
  • 5abd9a8: bump golang to 1.26.1 and update deps
  • 377fc61: fix: satisfy stricter Go 1.26.1 linter checks

2.24 - Silo 20260314 Released

Migration to the community-maintained Console fork and a large compatibility and dependency refresh.

Published: 2026-03-14 · Version: RELEASE.2026-03-14T12-00-00Z

This release switches the project to the community-maintained Console fork and performs a sizeable dependency refresh to establish the base for the later Go 1.26.x maintenance releases.

Major Changes

  • Switch to the community-maintained georgmangold/console v1.9.1 fork in place of the unmaintainable upstream Console dependency.
  • Refresh a large portion of the direct and indirect dependency graph so the new Console and toolchain combination builds cleanly.
  • Fix the go vet format directive issue in grid_test.go and adjust tests for the HTTP behavior changes in Go 1.26.

Fixed CVEs

  • CVE-2025-47913: golang.org/x/crypto v0.37.0 -> v0.46.0 fixes a panic in ssh/agent when handling malformed responses.
  • CVE-2025-58181: golang.org/x/crypto v0.37.0 -> v0.46.0 fixes unbounded memory consumption in ssh GSSAPI authentication parsing.
  • CVE-2025-47914: golang.org/x/crypto v0.37.0 -> v0.46.0 fixes a panic in ssh/agent caused by malformed identity messages.
  • CVE-2025-47911: golang.org/x/net v0.39.0 -> v0.48.0 fixes quadratic parsing complexity in html.Parse for crafted inputs.
  • CVE-2025-58190: golang.org/x/net v0.39.0 -> v0.48.0 fixes an infinite parsing loop in golang.org/x/net/html.
  • 68521b3: add github ci/cd pipeline
  • 00f3cf7: RELEASE.2026-03-14T12-00-00Z with go 1.26.0

2.25 - Silo 20260214 Released

Embedded Console restoration, GitHub CI/CD, Go 1.26.0, and the first complete community delivery surface.

Published: 2026-02-14 · Version: RELEASE.2026-02-14T12-00-00Z

This early infrastructure-focused community release restores the embedded Console, introduces GitHub CI/CD, and lifts the Go baseline to 1.26.0, which also absorbs a batch of security fixes from the older toolchain generation.

Major Changes

  • Restore the embedded Console and refresh the README to clarify the community fork position.
  • Add GitHub CI/CD workflows as the base for automated builds and multi-platform delivery.
  • Add quick links for docs, Docker, the GitHub repository, and installation through the pig package manager.

Fixed CVEs

These issues were absorbed as part of the Go 1.25.5 -> 1.26.0 upgrade:

  • CVE-2025-68121: crypto/tls could incorrectly accept mutated CA configuration during session resumption.
  • CVE-2025-61730: TLS 1.3 could process handshake messages incorrectly across encryption-level boundaries.
  • CVE-2025-61726: net/url query parsing could be abused for memory exhaustion.
  • CVE-2025-61728: archive/zip could consume excessive CPU while building archive indexes.
  • CVE-2025-68119: cmd/go could trigger unexpected code execution when invoking external VCS tooling.
  • CVE-2025-61731: the #cgo pkg-config: directive could be abused for arbitrary file writes.
  • CVE-2025-61732: cmd/cgo comment parsing discrepancies could enable code smuggling.
  • 8630937: Restore embedded console and update README for community fork
  • 68521b3: add github ci/cd pipeline

Historical note: CI/CD pipeline evidence now links maintained-history commit 68521b37f. It preserves the pipeline files and includes a README adjustment; this does not claim the full historical commit trees are identical.

2.26 - Silo 20251203 Released

The first community packaging and distribution baseline, with APK, DEB, and RPM artifacts.

Published: 2025-12-15 · Version: RELEASE.2025-12-03T12-00-00Z

This is the earliest traceable community release. Its purpose is to establish the community packaging and distribution baseline rather than to deliver incremental fixes over an earlier community release.

Major Changes

  • Build the community packaging flow around minio/pkger.
  • Choose a maintenance-mode upstream MinIO baseline as the starting point for the community-maintained fork.
  • Produce the first apk, deb, and rpm artifacts for ongoing community releases.

Fixed CVEs

  • This is the first community release. The GitHub Release does not provide a delta-style security-fix list against an earlier community version, and this page does not attempt to reconstruct the full historical CVE delta against the upstream maintenance baseline.
  • d4cd4b4: RELEASE.2025-12-03T12-00-00Z with go 1.25.5

3 - SILO Security Chronicle

CVEs, SILO-local SN advisories and correctness audits, with impact, evidence and release scope.

This is the security chronicle of the SILO community fork, listed from newest to oldest. Records may cover a CVE, a SILO-local SN finding, several related repairs, or a non-vulnerability correctness audit: the original threat model, the turns taken during review, the rejected alternatives, the final invariant, the evidence, and the compatibility cost all stay with that incident.

The security ledger indexes identifiers, fixed source and first containing releases. An investigation date is not a release date, and publishing an article does not deliver a fix.

3.1 - September Security Repairs: Payload Integrity, IAM Revocation and Console Sharing

SN-2026-012 through SN-2026-014, with distinct Server source and Console release boundaries.

This record covers three repairs and their delivery boundaries as of September 16. The identifiers are SILO-local advisory numbers, not CVEs or assigned CVSS scores. The security ledger is the maintained index.

Finding Fixed source Published delivery on September 16
SN-2026-012: presigned payload integrity Server c4b5e1cb4, PR #177 Not in Server RELEASE.2026-09-03T13-18-01Z; awaiting a later Server release
SN-2026-013: durable IAM revocation Server PR #191 / #192 Not in Server 20260903; coordinated upgrade required
SN-2026-014: anonymous Console share proxy Console PR #56; Server PR #209 selects the repair Standalone Console v2.4.1 is published; the embedded copy still requires a new Server binary

Presigned payload integrity — SN-2026-012

The generic authenticated path could verify a presigned request using a payload hash supplied only in X-Amz-Content-Sha256, yet omit checking the actual body against that hash. A holder of a suitably signed write URL could replace its body. The committed regression demonstrates this with PutBucketPolicy; this is not a claim that an anonymous caller can write arbitrary bucket policies.

c4b5e1cb4 in PR #177 binds body validation to the effective signed hash; mismatches fail with XAmzContentSHA256Mismatch (400). An explicit signed UNSIGNED-PAYLOAD retains its intended meaning. The adjacent signed-field fixes align policy conditions with authenticated values and remove the X-Amz-Signature-Age scratch header. They complement SN-2026-011; they do not establish complete unsigned-header coverage for streaming SigV4.

Review custom signing clients before upgrading. Query hash values take precedence over the fallback header, so changing an unsigned header must not change what policy evaluation or body validation sees.

Durable IAM revocation — SN-2026-013

Delayed replication events, parent recreation and lost deletion history could restore identities or grants an operator had revoked. PR #191 and PR #192 retain source revisions, deletion tombstones and signed parent-revocation boundaries. Older events and child credentials cannot silently cross a retained revocation boundary.

This is a persistent-state change. Upgrade all sites and all nodes sharing an IAM backend together; mixed old/new nodes and rolling downgrade are unsupported. mcli admin cluster iam export omits deletion history and is not a complete recovery backup. Preserve a tested full-backend recovery point and reconcile known earlier revocations. Tombstones have no TTL or automatic compaction.

The design record documents ordering, remaining group-membership limitations and metrics. Follow the upgrade and recovery procedure, including the password-policy pre-step. A revocation can commit and then return HTTP 500 if cleanup fails; the response alone does not prove that the old credential is valid.

Anonymous Console share proxy — SN-2026-014

Jiri Pejchal reported that the unauthenticated shared-download proxy accepted URLs beyond the intended object-download surface on its configured Server origin. This is a same-origin proxy-boundary issue; it is not evidence of arbitrary-host SSRF or bypass of the Server’s S3 authorization.

Console #56 constrains scheme, host and port, requires a valid bucket/object path, rejects system and traversal paths and operation-changing query selectors, disables redirects, and propagates caller cancellation. Accepted object URLs retain their original signed bytes.

Upgrade a standalone installation to Console v2.4.1. Server #209 selects the fixed Console source on main, but installing a separate Console cannot patch the UI and proxy compiled into Server 20260903. Until an appropriate Server artifact is available, limit exposure of the affected Console share endpoint according to the deployment.

Adjacent dependency hardening

Server main upgrades amqp091-go to v1.14.0 for GHSA-6c5v-hqjr-5xxp, fixed upstream in v1.13.0. A malicious AMQP peer can trigger excessive allocation; the SILO path requires a configured AMQP notification target. This Server change also awaits a release after 20260903. pkg v3.14.1 separately ships the JWX JSON encoding update. Component publication and Server dependency adoption are separate facts.

3.2 - SN-2026-011: Fix and Release Status

The signed-header fix is on Server main; the latest public Server remains affected.

Status on 2026-09-13: SN-2026-011 is fixed on Server main, starting with 123325430. The latest published Server, RELEASE.2026-09-03T13-18-01Z, and earlier public Server releases are affected. No new fixed Server release is established by the pkg v3.14.0 or mcli 20260913 publication.

A holder of a signed PUT request could add an unsigned x-amz-copy-source header and turn the permitted write into a copy of another object readable by the signing key. A destination that permits anonymous reads can expose those copied bytes. This affects both presigned and Authorization-header requests; the holder does not need the signing credentials.

The patch checks the received x-amz-* headers against the signed-header set, with the protocol’s explicit exceptions, before dispatching the requested operation. Follow-up request-signing and checksum regressions are documented in the signed-header review.

Operators must update the Server to source containing the fix or a future release that explicitly includes it. Restrict write-signing credentials to the required objects and avoid exposing unnecessary read grants or anonymous readable upload destinations while planning that update. Updating a client or Console alone does not remove the Server defect.

Reported by Oren Yomtov. The advisory ledger records SN-2026-011 and its source fix; a CVE was requested. Do not substitute a dependency scan’s clean reachability result for this application-level status. See the component matrix for released versus main source and the Server changelog for the remaining release contents.

3.3 - CVE-2025-62506: Session-Policy Privilege Escalation

The upstream fix and both regression groups were inherited by SILO unchanged; this ledger note maps the upstream PR to the fork commit and release evidence.

Status: Inherited and released
First Silo community release: RELEASE.2025-12-03T12-00-00Z
Upstream fixed release: RELEASE.2025-10-15T17-29-55Z
GitHub advisory: GHSA-jjjj-jwhf-8rgr
Upstream fix: minio/minio#21642

A service account or STS account with a restricted session policy could use an “own account” operation to create another service account without the restriction. The child account then inherited broader parent permissions, turning valid low-privilege credentials into a privilege-escalation path.

Silo did not need a separate backport. The upstream fix and its two regression groups were already in the history from which the community fork was released. This note records that inheritance without duplicating the tests or reconstructing the investigation already captured by the upstream advisory and PR.

Commit mapping

Role Repository Commit Evidence
Upstream remediation minio/minio c1a49490 Merge commit for PR #21642
Silo inheritance pgsty/silo c1a49490 The fork preserves the same commit object and SHA; the first Silo community release descends from it

The identical SHA is the mapping: this is inherited source history, not an independent Silo implementation with a merely similar patch. The advisory ledger keeps the same source-to-fork record.

Regression evidence

PR #21642 added two regression groups and runs each against root and non-root parents:

Both remain wired into the standard cmd test suites, so go test ./cmd executes them. No parallel or replacement regression test was added for this publication update.

Fix and operator boundary

When a session policy exists, the fix clears the DenyOnly shortcut before evaluating it. An “own account” exception therefore cannot turn “not explicitly denied” into permission: the restricted policy must actually allow the action.

Every Silo community release contains this commit. Operators migrating from a MinIO build older than the upstream fixed release should upgrade, review service accounts created by restricted service or STS identities, and revoke suspicious child accounts. This inheritance statement does not extend support to historical Silo releases; the current release line remains the supported line.

3.4 - SILO 20260903 Security Notes: SN-2026-006 through 010

Five inherited server authorization and encryption findings fixed by SILO 20260903: SSE-C key authentication, replication-header trust, admin status authorization, and explicit-version deletion.

Released in SILO 20260903. These fixes are part of RELEASE.2026-09-03T13-18-01Z. See the complete release notes for the 20260806-to-20260903 upgrade boundary, components, verification evidence, and known deferrals.

This bulletin collects five related security findings from the final SILO Server review. They cover customer-provided encryption keys, client-controlled replication headers, administrative status changes, and explicit object-version deletion.

The identifiers SN-2026-006 through SN-2026-010 are fork-local security-note IDs. They are not CVEs, are not registered in a vulnerability database, and should not be represented as such. The individual CVE chronicle remains one article per CVE; this release bulletin groups findings without CVEs so operators can evaluate one upgrade boundary.

All five defects were inherited from the archived minio/minio lineage and affect every earlier SILO release. “Inherited” describes provenance, not severity.

Summary

ID Required attacker position Security failure Released behavior
SN-2026-006 Read access to a zero-byte SSE-C object A wrong customer key was accepted because no data block was decrypted The object key is authenticated; wrong keys return 403 AccessDenied
SN-2026-007 Read access to an SSE-C object, or a client-supplied replication marker GetObjectAttributes returned protected metadata without authenticating the customer key Ordinary requests need the correct key; only an authorized replication peer uses the replica exception
SN-2026-008 Any authenticated principal able to invoke the affected read/write/delete operation Internal-looking headers granted replication-only effects without replication authorization Exact marker plus s3:ReplicateObject or s3:ReplicateDelete is required
SN-2026-009 Authenticated admin API principal holding only one status action The enable action authorized disable, and vice versa The requested target state selects the required admin action
SN-2026-010 Authenticated S3 principal with s3:DeleteObject but not s3:DeleteObjectVersion A caller could permanently delete an explicit version without the AWS-required action Explicit version deletion requires s3:DeleteObjectVersion

No finding in this bulletin is an unauthenticated remote-code-execution claim. Each requires an authenticated identity or existing object permission, but each crosses a privilege or cryptographic boundary beyond that identity’s intended grant.

SN-2026-006: zero-byte SSE-C key authentication

Affected operations: GetObject, HeadObject, CopyObject source processing, and GetObjectAttributes for zero-byte objects encrypted with SSE-C.
Tracking: issue #82.
Fixes: b73581b05 and c4fd97d0b.

SSE-C stores a sealed object key and requires the caller to provide the customer key again on reads. Normal reads authenticate that key while preparing the encrypted stream. A zero-byte object has no payload block, and the old fast path returned before unsealing the stored object key.

Consequently, an incorrect customer key could receive a successful response. A copy or new version could also be created under a key chosen by the caller without proving knowledge of the current key. The caller still needed read access to the object; the failure was that object permission replaced the separate cryptographic proof.

The fix authenticates the key independent of payload length. Correct keys behave as before. Wrong keys fail with 403 AccessDenied, including null-version and key-rotation cases.

SN-2026-007: SSE-C GetObjectAttributes

Affected operation: GetObjectAttributes on SSE-C objects.
Tracking: issue #84.
Fixes: 474cd5801, 74c97d005, and 21870fa2e.

GetObjectAttributes exposes object size, ETag, checksums, storage class, and multipart part metadata. The inherited handler returned those attributes without authenticating the supplied SSE-C key. It also treated the presence of X-Minio-Source-Replication-Request as permission to skip key handling.

The repair separates two cases:

  • an ordinary S3 caller must supply the correct SSE-C key;
  • a real replication peer may use the replica path only after authentication and s3:ReplicateObject authorization.

A wrong key now returns 403. A bare marker without the required key and permission cannot create the replica exception. Existing replication identities that already hold the required action keep working.

SN-2026-008: replication headers are not authority

Affected surface: object reads and writes, multipart completion, object and version deletion, Snowball extraction, Object Lock timestamps, checksum metadata, and bucket events.
Tracking: PR #101.
Primary fix range: 938603458 through 04b097fd9, followed by Snowball and rule-prefix hardening.

CVE-2026-34204 stopped ordinary PUT and COPY requests from importing a subset of replication SSE metadata. The wider audit found that many other consumers still treated header presence as proof that a request was internal replication.

Depending on the operation, a client holding ordinary read or write permission could:

  • request an SSE-C ciphertext/no-decryption path without the customer key;
  • preserve a supplied source ETag or modification time;
  • inject source checksum, actual-size, retention, legal-hold, or replica-state metadata;
  • choose replication-only delete semantics;
  • suppress successful object events;
  • carry trust from one Snowball archive entry into another.

The repair defines one receiver-wide authority:

  1. authenticate the original request in its signed form;
  2. require exactly one marker with the exact lowercase value true;
  3. authorize s3:ReplicateObject or s3:ReplicateDelete on the addressed resource;
  4. derive a narrower replica-trusted state only where replica status also proves it;
  5. store the decision in private request context;
  6. strip untrusted internal headers from a clone after authentication.

The context decision is authoritative. Stripping protects older consumers but cannot itself grant trust. The clone preserves the original trailer map so streaming checksum authentication remains correct.

The detailed protocol matrix, CORS interaction, black-box replication test, and rejected alternatives are in No I/O Before Auth, No Privilege From Headers.

SN-2026-009: user and group status authorization

Affected operations: admin SetUserStatus and SetGroupStatus.
Tracking: PR #73 / PR #85.
Fixes: 58735ee38 and 229fe2b3c.

The inherited handlers checked admin:EnableUser or admin:EnableGroup regardless of the target state. A narrowly delegated administrator permitted to enable an identity could therefore disable it too; an identity intended to hold only the disable action could fail or be evaluated against the wrong grant.

The requested state now determines the required action:

Target Required action
Enable user admin:EnableUser
Disable user admin:DisableUser
Enable group admin:EnableGroup
Disable group admin:DisableGroup

admin:* and the built-in consoleAdmin policy remain sufficient. Only custom least-privilege policies that accidentally relied on one action for both directions need adjustment.

SN-2026-010: explicit object-version deletion

Affected operations: DeleteObject and each DeleteObjects entry carrying an explicit versionId.
Tracking: issue #58 and PR #104.
Fix range: 75a6734e4 through d2d47a41f.

The inherited S3 path authorized explicit version deletion as ordinary s3:DeleteObject and used s3:DeleteObjectVersion only as a secondary deny check. This diverged from AWS and let a principal with ordinary delete permission permanently remove a selected historical version.

The release requires s3:DeleteObjectVersion for explicit versions. A delete without versionId continues to use s3:DeleteObject. Replication targets keep their s3:ReplicateDelete contract.

There are two policy effects:

  1. a principal granted only s3:DeleteObject can no longer delete explicit versions;
  2. a policy using Allow s3:* together with Deny s3:DeleteObject to block permanent deletion must also deny s3:DeleteObjectVersion.

The multi-delete repair preserves authentication and audit context for every entry and has least-privilege tests for ordinary and replication deletes.

Operator actions

Before upgrading:

  1. identify policies and applications that delete explicit versions; grant or deny s3:DeleteObjectVersion intentionally;
  2. inspect custom admin policies for enable/disable user and group operations;
  3. confirm that replication service accounts have only the required s3:ReplicateObject / s3:ReplicateDelete actions and that ordinary application identities do not;
  4. verify SSE-C integrations send the same customer key for GET, HEAD, attributes, and copy-source operations, including zero-byte objects;
  5. if an application was relying on a wrong SSE-C key returning success, treat that as a latent client defect and correct it before rollout.

After upgrading:

  • exercise a wrong-key negative test and a correct-key positive test on a disposable SSE-C object;
  • test one normal replication and one delete replication operation;
  • test each delegated admin direction independently;
  • test version deletion with a principal that has only s3:DeleteObject and with one that also has s3:DeleteObjectVersion;
  • review audit logs for unexpected denied operations instead of widening policies immediately.

Verification boundary

The fixes have focused unit, handler, authorization, streaming-trailer, Snowball, and site-replication tests. Full local acceptance on ebac0ca73 included the complete cmd race suite and six deployment shapes with 174 PASS / 0 FAIL. The final line adds one CORS state-clear fix after that run and passes targeted race, compatibility, generation, diff, and lint gates.

The final RELEASE.2026-09-03T13-18-01Z publication completed the gates recorded in the pre-release review: exact-SHA CI and Test Release, signed packages, checksums, SBOMs, provenance, and public classic/distroless container images. This bulletin still scopes its claims to the five fixes above and the residual risks below.

Residual risks outside this bulletin

  • Conditional deletion is still unsupported: DeleteObject ignores If-Match and DeleteObjects ignores per-object ETag (#10).
  • Non-CORS policy/SSE/tag/quota deletions may not converge across site-replication peers (#77).
  • The trust audit focused on known replication and SSE-C fields. New internal headers must still answer the same provenance-and-authorization question during review.

The release notes contain the full known-issue and deployment boundary.

3.5 - CVE-2026-32285: The jsonparser Advisory That Required No Patch

A security investigation that ended without a code change: the resolved dependency already contained the fix, and reachability analysis found no vulnerable path.

Status: Closed without a code change
GitHub issue: pgsty/minio#26

Security maintenance is not always a sequence of “find a vulnerability, then ship a patch.” The initial assessment of CVE-2026-32285 was that the repository might still carry a vulnerable jsonparser; replacing the dependency or maintaining another fork was even considered. Checking the resolved module version and actual reachability changed the conclusion: the tree already used v1.1.2, which contained the fix, and govulncheck found no reachable vulnerable symbol.

The right final action was not to manufacture an upgrade. It was to record the evidence and close the issue.

What was wrong with the initial assumption

The issue was first understood as “this dependency has no fixed version.” Acting on that premise without verification could have produced several changes that looked proactive but made the project worse:

  • changing the dependency graph for no security benefit;
  • introducing compatibility regressions in the name of a nonexistent fix;
  • adding another fork that would need long-term maintenance;
  • implying that previous SILO releases were demonstrably exposed when that had not been established.

Security work cannot be measured by whether it produces a diff. Leaving correct code unchanged is itself a security decision, and it needs evidence.

Investigation

The investigation narrowed the question through four layers of evidence:

  1. Confirm the version actually selected in the current go.mod and go.sum graph.
  2. Check the upstream release and establish that v1.1.2 already contained the relevant fix.
  3. Run and inspect govulncheck; it reported no reachable vulnerable symbol.
  4. Attribute the discrepancy in the issue to stale advisory or vulnerability-database information, not to a vulnerability still present in the source tree.

Four claims must remain separate: a version was once listed as affected, a package is imported, a vulnerable symbol is reachable in the program, and remote input can actually exploit that path. None of them proves the others.

Why there was no “just in case” upgrade

If the selected version already includes the fix, bumping to an arbitrary newer version does not make the system safer. It only expands the change surface and makes later regressions harder to attribute. That is especially risky in a large Go module graph.

The final decision was therefore to:

  • avoid committing a fictitious fix;
  • preserve the version and reachability evidence in the issue;
  • keep version gates and govulncheck in place to detect a future dependency rollback;
  • treat “no change required now” as a dated conclusion, not a permanent exemption.

Verification boundary

This incident established that the checkout examined on 2026-04-15 did not require a code change for CVE-2026-32285. It does not establish that every future branch, module graph, or release will remain unaffected. A dependency downgrade or a change in module selection requires the version and reachability checks to be repeated.

This article records the investigation and the basis for closure. The original govulncheck was not rerun while preparing this chronicle.

The principle this incident left behind

The objective of security maintenance is an accurate risk conclusion, not a patch for every issue. For a dependency CVE, ask in order: which version is actually selected, whether the vulnerable code enters the program, whether the symbol is reachable, and whether a deployed entry point makes it exploitable. Only when those answers require a source change should the investigation produce a diff.

3.6 - CVE-2026-33322: OIDC JWT Algorithm Confusion

The OIDC verifier mixed a client secret with JWKS keys; restoring asymmetric, JWKS-only verification closed the algorithm-confusion path.

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Affected entry points: AssumeRoleWithWebIdentity, AssumeRoleWithClientGrants
GitHub issue: pgsty/minio#22

The old implementation placed the OIDC client secret in the JWT verifier keyring while also accepting HMAC signing methods. An attacker who knew that client secret could therefore mint an HS-signed token and exchange it through STS for temporary credentials. The final fix restored asymmetric, JWKS-only verification. It deliberately broke HS256/384/512 compatibility instead of keeping an option that would reintroduce ambiguous trust semantics.

The vulnerability was not the absence of signature verification

At first glance, the old code did verify JWT signatures. The boundary that failed was more specific: which kind of key the verifier would accept, and whether the token header could select an algorithm with semantics that did not match that key’s intended role.

The attack chain required several conditions:

  • the attacker obtained the OIDC client secret;
  • the attacker constructed an HMAC-signed ID token;
  • the verifier treated the client secret as an HMAC signing key;
  • the token reached the WebIdentity or ClientGrants STS flow and was exchanged for temporary credentials.

Disclosure of a client secret is already serious, but it should not automatically confer the power to issue arbitrary user ID tokens. Combining those capabilities in one keyring created the algorithm-confusion vulnerability.

A compatibility path was implemented, then deliberately removed

During the fix, an allow_hmac-style compatibility path was implemented. It appeared reasonable: keep the secure default while letting users with a real requirement opt in. But retaining a shared secret in the general verifier keyring meant administrators would need to understand that the option expanded the entire STS trust boundary. Any future drift in the method allowlist could reopen the flaw.

The trade-off became clear:

Option Benefit Risk Decision
Keep the secret keyring and restrict some algorithms Small change; preserves HMAC IdPs The keyring still mixes two trust semantics Rejected
Add an allow_hmac option Makes compatibility explicit The option is difficult to reason about correctly and expands the test surface Implemented, then reverted
JWKS-only verification Clear boundary; refresh and retry use the same parser HS users must migrate Accepted

The most important decision was not what code was added, but that a completed compatibility implementation was removed.

Final invariants

The fix was concentrated in the OIDC JWT verification path and established four rules:

  • verifier keys come only from the identity provider’s JWKS;
  • the OIDC client secret never enters the JWT verification keyring;
  • HS256, HS384, and HS512 are always rejected;
  • the ordinary RS256 flow and JWKS refresh/retry use the same method allowlist.

The CVE was not used as a pretext for expanding JOSE support. PS256 and EdDSA remained out of scope.

Verification and release

The development record includes HS256 rejection, RS256 acceptance, JWKS refresh/retry regression tests, and focused go test ./internal/config/identity/openid. Temporary compatibility helpers, configuration, and tests were all removed from the final diff.

The public fix is d24f449, released with SILO 2026-04-17. This article records the historical verification; those tests were not rerun while preparing the chronicle.

Compatibility cost

This is an explicit breaking change. Identity providers that still issue HS256/384/512 tokens must migrate to JWKS-backed RSA or ECDSA before upgrading SILO. The project chose a narrower trust model that is easier to explain and audit over preserving a configuration that happened to work before.

3.7 - CVE-2026-33419: LDAP STS Enumeration and the Throttling Chain

From uniform authentication failures to corrected refunds, proxy attribution, and account lockout: LDAP STS hardening through two rounds of counter-fixes.

Status: Released, followed by two rounds of corrections
First containing release: RELEASE.2026-04-17T00-00-00Z
Complete correction: RELEASE.2026-06-18T00-00-00Z
GitHub issue: pgsty/minio#23

The core vulnerability was straightforward: LDAP STS returned different results for “user does not exist” and “password is wrong,” creating a username oracle. The first fix unified the external authentication failure and added limits by source IP and username. Continued review then showed that success refunds, spoofable source headers, reservation accounting, and the shared username bucket could turn the security control itself into a new attack surface.

The final June design removed the username bucket that enabled precise account lockout, retained only the source-IP bucket, and made proxy attribution an explicit deployment contract.

Initial threat model

The entry point is AssumeRoleWithLDAPIdentity. An attacker needs no existing MinIO account. Access to the LDAP STS endpoint is enough to compare the code, status, or message returned for an unknown user and a wrong password, enumerate valid usernames, and combine that knowledge with password spraying, guesses about organizational naming, or social engineering.

The fix could not simply disguise every error as “wrong password.” LDAP connection, lookup-bind, and directory-service failures still needed to appear as infrastructure errors, or operators would lose the ability to diagnose the service.

First round: uniform responses and a limiter

The initial fix on 2026-04-15 did three things:

  • unknown user and bad password returned the same external STS authentication error;
  • LDAP infrastructure failures still returned 500, with the real cause retained in the server log;
  • a new in-memory limiter initially created buckets for both source IP and normalized username.

This closed the content side channel and raised the cost of brute-force attempts, but the limiter state machine and source attribution exposed more problems.

Second round: success, attribution, and accounting

The follow-up changes on April 16 addressed three classes of defects:

  1. Successful authentication must not consume the failure allowance; the reserve/commit/cancel/refund lifecycle had to be explicit.
  2. The socket peer must be used by default; X-Forwarded-For, X-Real-IP, and Forwarded cannot be trusted merely because a request supplies them.
  3. Refund and capacity need hard bounds so cancel logic cannot mint tokens.

A proxy must be placed on an explicit trusted allowlist before it can influence source attribution.

Third round: remove the username bucket

Adversarial review in June overturned the intuition that “source plus username must be stronger than source alone.” A shared username bucket could be exhausted continuously from arbitrary origins. With only a low request rate, an attacker could lock a targeted account before the legitimate user ever reached an LDAP bind.

The final fix therefore:

  • removed the per-username bucket;
  • peeled trusted hops from XFF right to left and selected the first untrusted address;
  • rejected trusted-proxy footguns such as 0.0.0.0/0 and ::/0;
  • stopped using Forwarded for security-sensitive bucketing;
  • allowed X-Real-IP only under a contract in which the proxy overwrites rather than forwards client input.

This turn in the review shows that a security control needs its own threat model. More dimensions of throttling do not automatically mean more security.

Rejected alternatives

Option Why it was rejected
Perform a dummy bind for unknown users Amplifies LDAP load and creates a second, error-prone authentication path after the content side channel is already closed
Bucket all IPv6 clients by /64 Legitimate users behind the same site or carrier prefix can throttle one another
Take the leftmost XFF value Client-controlled and therefore spoofable
Fall back to the peer when XFF and X-Real-IP disagree An attacker can create disagreement deliberately and collapse every user behind a proxy into one bucket
Fully support RFC 7239 Forwarded Security-sensitive parsing complexity outweighs the practical benefit

Verification and release

The historical record covers limiter reserve/commit/cancel/refund behavior, concurrency, success and infrastructure failures, external equivalence of unknown-user and bad-password responses, and RemoteAddr, spoofed-header, trusted-proxy, multi-hop, and catch-all-CIDR cases. Focused package tests and builds were recorded as passing.

The LDAP security end-to-end test skips when _MINIO_LDAP_TEST_SERVER is absent, so an outer ok cannot be presented as proof of the full LDAP scenario.

The first public fix was 3b950f8. Follow-up corrections include 18b712d, 9e10f6d, f441108, and 5e40665.

Final cost and residual risk

  • The limiter now keys only on source IP and gives up a hard per-account throttle across different origins.
  • It is per-node and in-memory, not a cluster-wide password defense.
  • Botnets, distributed origins, IPv6 address rotation, and LDAP bind timing remain.
  • Incorrect trusted-proxy configuration can still destroy source attribution.
  • A Forwarded-only deployment falls back to the peer bucket and loses granularity.

Rate limiting can reduce attempts from one source. The uniform external authentication response is what actually conceals whether a username exists.

3.8 - CVE-2026-34204: Replication Metadata Injection

Ordinary PUT and COPY requests could forge internal replication state; the fix restores replication-only metadata solely inside authorized replication paths.

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
GitHub issue: pgsty/minio#24

Ordinary PUT and COPY requests could smuggle X-Minio-Replication-* headers into internal X-Minio-Internal-* SSE metadata, creating objects whose replication state did not match the authorized path and could even make them unreadable. The final fix stopped accepting replication-only metadata by default, restored it only in a trusted flow authorized for ReplicateObjectAction, and sanitized CopyObject before any header consumer ran.

Threat model

An attacker needed only ordinary object-write permission, not internode credentials. The input came entirely from client-controlled X-Minio-Replication-* headers, but metadata extraction converted it into internal replication or SSE state.

Later read paths interpreted the object according to that false internal state. The result could be an unreadable object: an integrity and availability failure. Almost every production server accepting untrusted writes needed to be treated as affected.

The root problem was not the header name. It was that data from an untrusted source acquired internal semantics without passing replication authorization.

Reject the whole request, or sanitize precisely?

Rejecting an ordinary request whenever it contains a replication header is the most obvious fix. It would also turn a header clients were previously allowed to send and have ignored into a hard failure. The final design was more precise:

  • the default extraction path does not accept replication-only metadata;
  • ordinary PUT and COPY strip those fields first;
  • only a path authorized for ReplicateObjectAction restores them;
  • replica-status writes use the same trusted condition;
  • legitimate multipart and Snowball replication flows explicitly restore the SSE metadata they require.

That keeps the compatibility change inside internal semantics instead of expanding it to every client carrying an extraneous header.

Why CopyObject had to sanitize early

CopyObject headers are not used only for the final metadata map. They can be consumed earlier by precondition logic and SSE-C source handling. Removing them immediately before the object write is too late: earlier consumers have already been contaminated.

The final sanitization occurs before those consumers. “Untrusted replication headers never enter internal semantics” becomes one invariant instead of a convention every downstream function must remember to enforce.

Implementation and verification

The change covered handler utilities, object handlers, and multipart handlers, with tests at several layers:

  • trusted and untrusted metadata extraction at the helper layer;
  • malicious PUT and COPY cases at the handler layer;
  • CopyObject header sanitization;
  • red/green comparison between the vulnerable parent and the patched tree;
  • live-server before/after behavior showing that a malicious header no longer made an object unreadable;
  • continued operation of legitimate replication, multipart, and Snowball flows.

The public fix is 56fa63b. This article preserves the historical verification boundary; no live server was started again while preparing the chronicle.

Cost and residual risk

  • Internal replication headers supplied by ordinary clients are now ignored or stripped.
  • Replication-only metadata must be restored explicitly inside an authorized branch.
  • If a future replication entry point forgets to restore it, the result should be a functional regression rather than another untrusted write path.
  • The audit focused on replication headers; it does not establish that every X-Minio-Internal-* field has undergone the same trust review.

This incident leaves a simple review question: a field that looks “internal” is not necessarily trusted. Ask where it came from and which authorization decision allowed it to acquire internal meaning.

The later CORS and replication trust design record generalizes that lesson into one receiver-wide authority for SSE-C reads, source timestamps, multipart completion, object lock, deletes, events, and pre-authentication CORS work.

3.9 - CVE-2026-39414: Oversized S3 Select Records and a SIMD Bypass

The first fix imposed a 1 MiB bound on CSV and JSON Lines; the second found that the SIMD fast path bypassed it completely.

Status: Released; the second-round fix was completed in June
Initial fix release: RELEASE.2026-04-17T00-00-00Z
Complete fix release: RELEASE.2026-06-18T00-00-00Z
GitHub issue: pgsty/minio#25

The first fix in April reused the existing 1 MiB maxCharsPerRecord limit for both CSV and ordinary JSON Lines. This prevented unbounded buffering while waiting for a delimiter and returned the explicit OverMaxRecordSize error to clients. A June review then found that CPUs with SIMD support took a different simdjson fast path that bypassed the limit completely.

The final solution sent JSON Lines through the bounded reader on every CPU. The same review also corrected error mapping, parser errors, and the flush of completed records before a terminal error. SILO temporarily gave up the SIMD fast path in exchange for consistent security semantics.

Threat model

An attacker can submit or query an object containing an extremely long single record. The reader continues buffering until it sees a record delimiter, allowing memory and CPU denial of service. More subtly, the same input can select a different implementation according to the machine’s CPU features. Safe behavior on a test machine does not necessarily prove safe behavior in production.

Error semantics are part of the fix. If an oversized record appears only as a generic InternalError, clients and monitoring systems cannot distinguish an enforced security limit from a server failure.

First round: reuse the existing 1 MiB invariant

The first patch did not invent a new configuration knob. It applied the existing maxCharsPerRecord = 1 MiB rule:

  • the CSV splitter and line-delimited JSON rejected oversized records before buffering or parsing them further;
  • the earliest splitter error was preserved instead of being overwritten by a partial decode;
  • the error propagated as OverMaxRecordSize rather than collapsing into InternalError.

This was a deliberate compatibility contraction. Clients with lines or records larger than 1 MiB now had to split their input.

Second round: a hardware-dependent bypass

Following the call chain again in June exposed this path:

JSON Lines -> simdj.NewReader -> simdjson.ParseNDStream

When simdjson.SupportedCPU() returned true, JSON Lines bypassed the bounded json.PReader. The third-party parser kept reading past a chunk boundary until it found a newline. A generic reader wrapper could not simultaneously preserve already completed records and guarantee a bound on the next record.

The final choice was not another wrapper. JSON Lines temporarily stopped using the SIMD path and always used the bounded PReader. If SIMD support returns, that implementation must enforce the same record limit itself and pass the same CPU-independent regression suite.

Stream semantics corrected in the same round

The review also fixed several adjacent behaviors:

  • use errors.As to pass through errors implementing SelectError, not just one concrete type;
  • have the JSON worker wrap parser failures as JSONParsingError;
  • flush completed records still waiting below the batch threshold before emitting a terminal error event;
  • preserve error priority in input order instead of letting a later oversized record overwrite an earlier parse error.

Those details determine whether a client sees the correct failure or a resource-limit fix that quietly broke the streaming protocol.

Deliberately left outside this CVE

  • The historical mismatch between CSV AllowQuotedRecordDelimiter and the outer physical-newline splitter.
  • Whether \r in CRLF counts toward the record length.
  • Restoring SIMD performance without an equivalent bound.

These questions may be real, but they require independent AWS-compatibility evidence or a more complex quote-aware splitter. They did not belong in a security patch based on guesses.

Verification and release

The historical record includes oversized JSON Lines, error-code preservation, and behavior tests that do not depend on the local machine’s SIMD capabilities. go test ./internal/s3select/... -count=1 and git diff --check were recorded as passing.

The initial public fix was 3252d5b; the complete June correction is fd69c89. Those tests were not rerun while preparing this article.

Final trade-offs

  • JSON Lines performance may decrease; this incident did not produce a benchmark that quantifies it.
  • The 1 MiB per-record limit rejects oversized input accepted by previous releases.
  • Quoted, multiline CSV semantics still need separate work.
  • Any future CPU-specific fast path must share the slow path’s security tests.

The second fix leaves the central lesson: a security invariant must hold across hardware-dependent paths. A green test on one CPU does not prove that another execution engine is protected.

3.10 - CVE-2026-40344: Snowball Auto-Extract Authentication Bypass

A Snowball unsigned-trailer request could reach the extractor before authentication; the fix verifies SigV4 before any tar byte crosses that boundary.

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
GitHub advisory: GHSA-9c4q-hq6p-c237

Snowball’s PutObjectExtractHandler omitted the streaming unsigned-trailer authentication case. A tar stream with a forged signature could enter untar() before authentication completed, and one request could fan out into many object writes. The final fix initialized the correct reader, handled the decoded length, and completed SigV4 verification before any tar byte reached the extractor.

Why the identifier changed

The official CVE had not been assigned when the fix was written, so the commit subject used the temporary identifier fake CVE-2026-40028. The final identifier is CVE-2026-40344. The historical commit was not rewritten; the advisory and this article use the official number.

From one missing authentication case to bulk object writes

The entry point was Snowball / PutObjectExtract auto-extraction. The request used unsigned-trailer streaming, an authentication type the old handler did not cover as ordinary PUT did.

The danger was larger than one incorrectly authorized request. Once the tar stream entered untar(), that request could create multiple attacker-chosen objects. An authentication omission was therefore amplified into a bulk-write problem.

Final invariant: the extractor sees zero bytes on failure

The key statement in the fix was:

If authentication ultimately fails, untar() must have seen zero bytes.

That rule excludes “extract first, then roll back if authentication fails.” Object writes travel through several paths, and proving a complete rollback is much harder than proving that input never crossed the boundary. Authentication had to close before data entered the extractor.

Implementation

The final change:

  • recognized authTypeStreamingUnsignedTrailer;
  • read X-Amz-Decoded-Content-Length;
  • used newUnsignedV4ChunkedReader();
  • performed complete SigV4 request verification before entering untar();
  • preserved valid signed Snowball requests and CRC32 trailer flows.

Verification

The historical commit and investigation record cover:

  • rejection of a forged-signature Snowball unsigned-trailer request;
  • rejection of anonymous Snowball writes to a non-public bucket;
  • successful extraction with a valid signature and trailing CRC32;
  • red/green comparison between the vulnerable parent and the patched tree;
  • containerized before/after smoke tests.

The public fix is efb6e5b00. The container tests were not rerun while preparing this article.

Compatibility and residual risk

  • Clients that relied on an unsigned-trailer Snowball combination whose signature was never really verified will fail after upgrading.
  • Authentication now closes before extraction, but tar paths, archive-size limits, and object-count limits remain separate security surfaces.
  • Snowball and ordinary unsigned-trailer requests now share a reader; future changes must regress both paths together.

The essence of the fix was not another if. It moved the authentication decision in front of the actual amplification boundary.

3.11 - CVE-2026-41145: Unsigned-Trailer Query Authentication Bypass

Query-string SigV4 credentials entered the unsigned-trailer stream without signature verification; the shared reader now closes that boundary once for every caller.

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
GitHub advisory: GHSA-hv4r-mvr4-25vw

Query-string SigV4 credentials could enter a STREAMING-UNSIGNED-PAYLOAD-TRAILER data flow, while the old code verified the signature only when an Authorization header was present. A request carrying a valid access-key identifier could therefore complete a write without a correct signature.

The final fix moved presigned rejection and SigV4 verification into newUnsignedV4ChunkedReader(), making every caller consuming that stream share one authentication boundary.

Identifier note

The official CVE had not been assigned when the patch was written, so its commit subject used fake CVE-2026-40027. The final identifier is CVE-2026-41145. The historical commit remains unchanged; public material uses the official identifier.

Root cause: authentication was coupled to transport form

The affected entry points included PutObject and PutObjectPart. The request selected STREAMING-UNSIGNED-PAYLOAD-TRAILER, with its credentials and signature in the query string rather than the Authorization header.

The old handler used header presence to decide whether to verify a signature. The body reader still consumed the data normally, silently degrading query authentication into something close to an anonymous write. The attacker needed to know a valid access-key identifier, but did not need to produce a correct signature.

The problem was not failure to parse the query parameters. It was that authentication depended on how credentials were transported instead of the trust boundary at which the stream was consumed.

Why the patch did not live in each handler

Option Risk Decision
Add header/query checks separately to PutObject and PutObjectPart Closes today’s entry points, but a new caller can omit the check again Rejected
Invent a compatible presigned unsigned-trailer protocol Greatly expands protocol and test surface without an existing support contract Rejected
Reject and verify centrally in newUnsignedV4ChunkedReader() Forces every consumer through the same boundary Accepted

Anonymous unsigned-trailer requests were not prohibited wholesale. If bucket policy explicitly permits anonymous writes, they can still follow the anonymous authorization path. The forbidden state is the mixture of query credentials with no verification of those credentials.

Implementation and verification

The fix performs presigned rejection and SigV4 verification at the reader entry in cmd/streaming-v4-unsigned.go, while removing the gates in the PutObject and multipart handlers that depended on header presence.

New tests cover forged query PUT, multipart, mixed authentication, and anonymous policy. The historical record also includes a vulnerable-parent write that succeeded, failure on the patched tree, and live-server before/after smoke tests showing that header-authenticated and valid anonymous flows continued to work.

The public fix is f444b6f37. The live exploit was not rerun while preparing this article.

Compatibility and residual risk

  • Presigned/query unsigned-trailer is now explicitly unsupported, an intentional breaking change.
  • Moving the fix into the reader significantly reduces the chance that a sibling handler omits the check again.
  • Other streaming authentication modes still need their own audits; this reader fix does not establish that every SigV4 streaming combination is safe.

The shape of this fix matters as much as its payload: when several handlers share an authenticated data stream, authentication belongs to the reader rather than to optional checks in each caller.

3.12 - CVE-2026-42600: ReadMultiple Storage-REST Path Traversal

From complete preflight validation to deleting the API: why an internal file-reading endpoint with no production callers was not worth retaining.

Status: Released
First containing release: RELEASE.2026-06-18T00-00-00Z
GitHub advisory: GHSA-xh8f-g2qw-gcm7
Affected scope: Distributed erasure only; cluster-root / internode JWT required

The msgpack body of /rmpl carried Bucket, Prefix, and Files. The old code joined those values into filesystem paths without a containment check. The initial fix implemented full preflight validation. Continued call-chain review then found that this API had had no production caller since 2024. The final solution changed from “retain and harden” to removing the route, handler, client, interface, and generated code.

Deleting roughly a thousand lines was a larger source diff than a local validation guard, but it left a smaller long-term attack surface.

Threat model

The vulnerable route was registered only in distributed erasure mode; single-node deployments were unaffected. An attacker needed an internode JWT derived from the root secret, control of a node, or the ability to intercept unencrypted traffic between nodes.

The dangerous fields were inside the msgpack body, not the URL or form data, so upper HTTP path middleware never saw them. xlStorage.ReadMultiple joined and read the resulting paths directly, allowing them to escape the drive root.

This was not an anonymous S3 vulnerability. It crossed the boundary from “cluster root or peer” to “any file readable by the node process.”

First design: retain the API and validate it completely

The initial patch in xlStorage.ReadMultiple:

  • rejected absolute paths, . and .. segments, backslashes, Windows drive prefixes, and NUL bytes;
  • checked final containment across drive, volume, prefix, and file;
  • preserved the historical contract for an empty Bucket and .minio.sys/multipart where possible;
  • returned an error before any read or streaming began.

That design could close the known traversal, but review quickly exposed an early-return gap.

MaxResults exposed the danger of validate-as-you-use

The first patch validated each Files item inside the read loop. For Files=[good, bad] with MaxResults=1, the function returned after reading the first item and never validated the second.

It did not read the second malicious file, but it violated the intended invariant that the entire msgpack request must be valid before execution. Validation was therefore moved into a complete preflight pass, with path-length checks also completed before streaming.

The general rule is useful beyond this endpoint: when a request can return early or stream a partial response, checking an element immediately before use is not equivalent to validating the whole request.

Final decision: delete the API

Further call-chain audit established that:

  • upstream removed the last production caller in September 2024;
  • multipart had moved to ReadParts;
  • the current tree had no in-tree production consumer;
  • upstream’s final remediation also deleted ReadMultiple.

The final change removed the route, handler, client wrapper, StorageAPI / xlStorage method, metric, datatype, and generated code. storageRESTVersion retained the existing compatibility strategy.

Option Short-term change Long-term maintenance surface Decision
Validate in place Smaller diff and preserved endpoint Permanently retains an unused, privileged file-reading API Abandoned
Delete the API Removes more interface and generated code Minimizes attack and maintenance surface Accepted

Verification and release

The in-place validation phase ran focused tests for xlStorage, the storage-REST client, msgpack encode/decode, and path edge cases; adversarial review exposed the MaxResults flaw. The deletion phase checked the route, client, interface, generated surfaces, and absence of callers.

The public fix is 73ac524, released with SILO 2026-06-18. The post-deletion full suite was not rerun while preparing this article.

Compatibility and claim boundary

  • The external S3 API is unchanged.
  • Third-party implementations that privately called the internal /rmpl endpoint will stop working.
  • A mixed-version rolling upgrade may encounter a protocol mismatch, so cluster nodes should be kept on the same version during the upgrade.
  • Removing this endpoint proves only that ReadMultiple no longer exists; it does not establish that every internal node request carrying body paths has completed a containment audit.

This CVE reached the right final fix, but it also leaves an important distinction: closing one endpoint is not the same claim as closing an entire vulnerability class.

3.13 - Internode Path Containment Audit: Paying Off What CVE-2026-42600 Left Owing

The previous entry said plainly that deleting an endpoint is not the same as closing a defect class. This is that audit: four protocol surfaces, twelve defects, and four regressions we caused ourselves while fixing them.

Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work. Affected scope: Distributed erasure only; cluster-root / internode JWT required Prerequisite reading: CVE-2026-42600 · ReadMultiple

This article is now public. The primary fix shipped as recorded above; residual findings below are the investigation-time record, not a claim that publishing one fix closed the entire defect class.

The previous entry closed with this sentence:

Deleting the endpoint proves only that ReadMultiple no longer exists. It cannot be extrapolated into a completed containment audit of every internode body path.

That was an IOU, written down in plain sight. This is the record of paying it — and a not-very-flattering construction log.

Conclusions first

  • Twelve defects, all inherited from upstream. Verified by per-function md5 comparison: the fork’s diff against upstream on the affected files is pure deletion, zero added lines.
  • This is not a new vulnerability. It is the remainder of CVE-2026-42600 — three more protocol surfaces under the same root cause.
  • Our failure is not in the code. It is in the record: a point fix was written up as a closure.
  • While fixing it we introduced four regressions of our own, every one of them in a rule we invented rather than reused.

“N endpoints” was the wrong frame

Earlier audits kept counting endpoints, arriving at 19, then 21, then 22 — and missing an entire protocol surface each time. The real structure is four surfaces:

Protocol surface Entries Covered by the global HTTP middleware
storage-REST HTTP query arguments 9 Yes — previously misreported as unprotected
storage-REST HTTP msgpack body 4 No; r.Form never comes from a body
storage Grid RPC 18 No; after one upgrade, frames never re-enter the HTTP chain
peer-S3 Grid RPC 5 No, and it bypasses getStorage() to reach drives directly

The first row matters as much as the rest: it overturns the earlier “all 21 endpoints escapable” claim. Those audits grepped for the validation helper inside handler bodies, found nothing, and concluded there was no protection — missing that the protection lives in the middleware layer.

The fourth row is the one no amount of hardening in storage-REST handlers can reach.

The root cause is three layers, not one bug

Three design facts, none wrong on its own:

  1. Validation happens only at the HTTP surface. r.Form is populated from url.ParseQuery(RawQuery) and never from a body (introduced 2017).
  2. Grid RPC bypasses the middleware. /minio/grid/v1 upgrades once; subsequent msgpack frames never re-enter the HTTP chain (introduced 2023).
  3. The storage layer performs no containment. getVolDir rejects a volume only when it is exactly ""/./.., and pathJoin runs Clean (settled 2018).

In one sentence: the upper layer assumes the lower one validates, the lower assumes the upper already did, and neither can see the channel in between.

ReadMultiple was merely one endpoint that exercised that structure. Removing it left the structure intact.

One line of the timeline deserves singling out. The divide-by-zero in ShardFileSize has been present since 2020, but only when it moved inside xioutil.WithDeadline in 2024-10 — a change meant to fix large-object timeouts — did it escalate from “one failed request” to “the whole process exits”, because WithDeadline runs its work function on a bare goroutine that no recover() can reach. That escalation was not visible at the time.

Vectors, confirmed by execution

Every one reproduced against a real xlStorage through the real REST/grid client, with planted sentinel files. None of this is static inference.

Vector Surface Observed
WriteAll("vol","../../x") storage grid arbitrary file write outside the drive root
RenameFile(".minio.sys","","bucket","x") storage grid the entire system volume (IAM, config) relocated into a readable bucket, with no .. anywhere
DeleteBucket("../victim", force) peer-S3 grid recursive deletion of a tree outside the drive root
DeleteBulk("vol","") HTTP body whole volume moved to trash
ReadAll(volume:"../") any getVolDir’s check defeated by a trailing slash
CheckParts with a zero Erasure storage grid process terminates
AppendFile declaring Content-Length: 64 GiB HTTP 68,719,574,840 bytes allocated for an empty body
DeleteVersions declaring 100M entries HTTP a ten-byte parameter reserved 10.4 GB
part Size = -2 storage grid a truncated shard reported healthy; heal silently skipped

Two of these had never been found before and are worth calling out.

RenameFile with an empty source path hits the volume-root alias and relocates the whole volume. Aimed at .minio.sys, one ordinary S3 GET afterwards yields the cluster’s IAM and configuration. It requires no traversal sequence at all — so any audit that greps for .. misses it by construction.

A negative part size floors both terms of ShardFileSize to zero. checkPart’s only integrity test is st.Size() < expectedSize, so every file that exists is reported intact, including a truncated shard. Worse, this holds whether or not the erasure parameters are valid: metadata that passes FileInfo.IsValid() — the very check healing trusts — is affected. That is not an input-validation problem but a data-integrity one: a legitimate heal reading poisoned metadata concludes the shard is fine and skips the repair.

The fix: two chokepoints, not twenty patches

The invariant to restore is one sentence: a path from an internode payload must resolve inside the volume it names, and a volume must resolve inside the drive root.

Only .. can break the first half (absolute and backslash-prefixed paths are folded under volumeDir by pathJoin’s Clean), and the second half has one independent break: paths that alias the volume root. Two rules, therefore — not a policy matrix.

Chokepoint Location Coverage
Volume axis getVolDir (4 lines) every caller, including peer-S3
Path axis decorator at getStorage() 31 remote entries plus nested fields

Under 40 lines of core logic. No handler is modified, no call site in xl-storage.go is touched, and the local erasure path is left alone.

Two details worth recording:

  • The check must run before the join, on the raw argument. pathJoin runs Clean against an absolute drivePath, which erases a leading .. entirely — /drive/../../etc becomes /etc, so a check placed after the join reads clean and passes everything. This is the most likely way a future refactor silently undoes the fix.
  • NSScanner is the one method that reaches the filesystem without getVolDir. Its guard line is load-bearing, not decorative.

Among the rejected alternatives, the notable one is adding containment at all 33 pathJoin(volumeDir, …) sinks. That would be genuine defence in depth, but it means 33 edits in the most performance-sensitive file in the tree, each needing its own judgement about whether the volume root is a legitimate target. The guard rails buy most of the same resistance to drift for a fraction of the risk. This is an explicitly recorded IOU: if a code path is ever added that reaches the filesystem without getVolDir, the decision must be revisited.

Construction log: four regressions we caused ourselves

This section is unflattering and more informative than the fix.

First: whitespace. The initial rule treated whitespace as a separator, refusing " " and " "legal S3 object keys. PutObject commits through RenameData, so such a key would fail on every remote drive simultaneously and break write quorum. The irony: the vulnerability needs root credentials; this bug needs a user to send a space.

Second: backslash-only keys. Same function, same root cause. path.Clean never treats \ as a separator, so on Unix "\\" is an ordinary filename. Refusing it made a distributed cluster reject a write a single-node server accepts — the same S3 API behaving differently by deployment topology.

Third: spaces and periods on Windows. The Win32 normalisation layer strips trailing spaces and periods from a path component, so a component made only of those vanishes and the path resolves to its parent. That makes both " " and "..." volume-root aliases on Windows — and "..." was sitting in our own list of legal object names at the time. We had not merely missed the vector; we had asserted it was safe.

Fourth: negative part sizes. The new guard rejected only “positive size with unusable parameters”, equating “non-zero” with “positive”. Negative values take a different route to the same zero.

Two false greens

Writing the AppendFile acceptance test produced two meaningless green runs in a row:

  1. Driving it through the REST client — which special-cases *bytes.Reader and derives Content-Length from it, silently overriding the forged value.
  2. Switching to an opaque reader — at which point Go’s own HTTP client refuses to send a request whose body is shorter than the declared length.

Only driving the handler directly through httptest reproduced it. The lesson: a client’s self-protection is not a server’s defence, and an attacker with a raw socket has no such scruples.

A third was a design failure. We built a per-field reflection poisoner, then discarded it: it cannot distinguish “should have rejected but delegated” from “correctly allowed a non-path field” (ETag, Algorithm, …), so it reports correct behaviour as failure.

The subtlest lived in the fuzzer. The first property test treated separator-only strings as an exception with an early return. That is not an exception, it is a blind spot — the fuzzer had been shut out of the entire category by hand and could never have found the backslash key in a million executions. A wrong exception is more dangerous than no fuzzer at all, because it creates the impression the space has been searched.

A very concentrated pattern

Component Where its semantics came from Regressions
guardPaths reused existing hasBadPathComponent 0
getVolDir guard reused existing hasBadPathComponent 0
isVolumeRootAlias invented 3
guardErasureParams invented 1

The reused semantics produced zero regressions; the invented rules produced all of them.

This is not coincidence. hasBadPathComponent is already the object layer’s own rule via IsValidObjectPrefix, validated by real S3 traffic for years, and structurally cannot reject anything creatable through the S3 API. An invented rule has nothing behind it but the author’s imagination.

The actionable form: reuse rather than invent; and when you must invent, write the property test by exclusion rather than enumeration, express exceptions with a predicate independent of the implementation, and keep them as few as possible.

The guard rails matter more than the patch

The final test suite pulls in two directions, and neither alone is enough:

  • Falsifiability — remove each guard in turn and confirm the tests actually go red (191 failing subtests with the traversal guards removed; 64 GiB and 10.4 GB reappearing with the allocation guards removed). This is precisely what the rejected community PR lacked: its test asserted err != nil against a target that did not exist, so it passes with the vulnerability fully intact.
  • Legal-traffic fuzzing — asserting that any key IsValidObjectName accepts, the guards accept (1.96M executions, no violations), and that any legal bucket name survives getVolDir (810K). This is what our own first two attempts lacked.

Plus a method-level reflection rail that fails by name when a path-taking method is added to StorageAPI unguarded.

History states the case for these rails bluntly: CVE-2026-39414 was also point-fixed on 2026-04-15 and only received a fix: complete ... two months later. Counting this one, “point fix → recorded as closure → completed months later” has now happened twice in this fork. The problem is not that someone was careless. It is that nothing in the tree could tell you a class was still open. Guard rails turn “someone must remember” into “CI fails”.

On adversarial review

This fix went through five rounds of independent adversarial review. Each round found one missed defect, and all five stood: whitespace keys → backslash keys → the AppendFile allocation → Windows spaces and periods → negative part sizes.

Our own review did find two in the same period (the WithDeadline log amplification and ReadParts using the wrong rule), but only after being pushed to that standard.

The hit rate says something plain: the last gate before merge should be independent acceptance, not the author’s own conclusion. During this work the author judged the change ready to ship four times and was overturned three.

Follow-up status

The following five follow-ups were not published at the original August 3 review point. They subsequently shipped in Server 20260804:

  • ReadFileHandler is bounded. Commit b6f70ab08 rejects a declared read length above 5 GiB, the maximum size of the S3 part represented by this legacy whole-file bitrot path. Legitimate GiB-scale reads can still allocate on that scale; the change removes caller-controlled allocation above the format’s real ceiling rather than pretending large reads are cheap.
  • Negative part sizes cannot be persisted or trusted. Commit 80e8eaa42 rejects them at the AddVersion write funnel and again in CheckParts and VerifyFile, so both new poison and already-written metadata are covered. The internode boundary check uses the same predicate.
  • Non-positive erasure block sizes are rejected at construction. Commit 80e8eaa42 validates blockSize in NewErasure, covering the other offset and decode divisions that a single downstream ShardFileSize guard could not. Rebalance’s separate division is guarded at its own boundary.
  • ReadParts errors finally reach the caller (1af351a70). A pre-existing bug, fixed in passing: the handler declared success (done(nil)) before the storage call finished, so every ReadParts failure — not only the new guards, but errFileNotFound and friends too — was never returned; the client received a keep-alive stream and decoded the error text as msgpack, surfacing as msgp: attempted to decode type "int" with method for "map". After the fix, real storage errors reach quorum reduction; success paths are untouched, and that msgpack decode noise in multipart-related logs is replaced by actual storage errors.
  • A recovered panic is not a healthy node (a36fd8fff). The WithDeadline recovery turns what used to be an instant process death into a stderr line (panic in deadline-bounded work: ... with a stack) plus one failed request. Stack dumps are capped at ten per process lifetime, but the error returns every time. If your alerting only watches process liveness, add a log alert for that string — otherwise a real bug that previously announced itself as a crash now only whispers in the logs.

Two limitations remain and should not be folded into a stronger claim:

  • No Windows CI. Windows builds are published; tests run on Ubuntu only. The Windows rule is reasoned from documented Win32 behaviour and has not been verified on the platform.
  • Symlinks. The containment check is lexical, as upstream’s is.

Closing

The previous entry said that closing an endpoint and closing a defect class are two different conclusions. This time the known sinks are closed on the local branch, at the cost of four self-inflicted regressions and three overturned declarations that it was ready to ship. At the investigation date publication remained a separate gate; these fixes subsequently shipped in 20260804.

If only one sentence survives: the vulnerability was upstream’s; our mistake was treating a point fix as a closure. And what prevents a third occurrence is not a more careful person — it is a test that fails.

3.14 - The Parser Knew, the Schema Didn't: Config Keys That Could Take Every Notification Down

NATS JWT credentials were rejected as an invalid key by the very server that reads them. The same gap was wired into the legacy migration, so an upgraded config failed validation at every boot — and one failing subsystem zeroes the whole notification list. Three upstream feature PRs each forgot the same registration; a fourth surface corrupted values silently.

Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work. Classification: Configuration-schema consistency and availability, not a vulnerability; includes one defensive hardening (credential values no longer echoed in validation errors) Affected scope: notify_nats JWT/NKey/TLS-handshake-first options, notify_amqp immediate, and any pre-2020 config migrated with an enabled NATS target — whose failure then silences every notification backend Tracking: pgsty/minio issue #39

This article names two unfixed availability defects in neighbouring code (the Postgres/MySQL migration writes, and kvFields typo folding). Neither is exploitable — both break the operator’s own configuration, loudly or not at all — and both are already named in the committed audit test’s allowlist. Publication needs no hold beyond the release itself.

Conclusions first

  • Three notify_nats options — user_credentials, nkey_seed, tls_handshake_first — and one notify_amqp option — immediate — were read by the parser, written by the legacy migration, and registered nowhere. CheckValidKeys rejected exactly what GetNotifyNATS required.
  • One constant meant two things. target.NATSUserCredentials held the string "MINIO_NOTIFY_NATS_USER_CREDENTIALS", sat in the environment-variable const block, and was used both as an env var name and as a config key. The snake_case config key for creds-file auth did not exist anywhere in the program.
  • The reporter’s error did not come from their command. It came from the legacy migration: the pre-fix migration reproduces the issue’s error text byte for byte, including the notify_nats:ONE target name their command never mentioned. The migration wrote the store once; validation rejects it at every boot thereafter.
  • The blast radius is the amplifier: FetchEnabledTargets fails fast on the first bad subsystem, its only caller just logs, and the global target list stays nil — so one broken NATS entry silently switches off Kafka, webhook, MQTT, and everything else.
  • Inherited from upstream. Three feature PRs — #19139 (2024-02, user_credentials), #21008 (2025-04, tls_handshake_first), #21231 (2025-04, nkey_seed) — each added the parser and the env var, and each skipped the schema. Upstream is archived; the fork inherits both the defect and the duty.
  • The fix registers the keys, splits the two-faced constant, corrects the migration — including a sibling bug that silently wrote immediate’s value under the internal key — tolerates the legacy on-disk spelling on the load path only, stops echoing values in invalid-key errors in both CheckValidKeys forms, and installs an AST audit that mechanically forbids this defect class across all ten notify subsystems.
  • The audit found the next instance before the ink dried: the Postgres/MySQL legacy migrations write five unregistered keys, one of which is a plaintext database password. Recorded, allowlisted shrink-only, tracked for follow-up.

The error that named a target nobody asked about

The report (issue #39, by kuldeep-link11, against a NATS cluster using JWT operator/accounts auth) is a clean reproduction: configure notify_nats with a credentials file, watch it bounce.

$ mc admin config set us notify_nats:FITCHECK \
    address=nats-1:4222 subject=events.object.created \
    MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/creds/minio_notifier.creds \
    jetstream=off queue_dir=/data/queue-fitcheck queue_limit=100000

mc: <ERROR> ... found invalid keys
    (MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/creds/minio_notifier.creds
     nkey_seed= tls_handshake_first=off ) for 'notify_nats:ONE' sub-system,
    use 'mc admin config reset myminio notify_nats:ONE' to fix invalid keys

Two things in that error are wrong in ways the command cannot explain. The invalid-key list contains nkey_seed= and tls_handshake_first=off — keys the user never passed. And the rejected sub-system is notify_nats:ONE, while the command configured notify_nats:FITCHECK.

The second oddity is the whole case. Our reviewer established that the mc admin config set path cannot even carry an unregistered key: the server-side tokenizer, kvFields, splits the input line by searching for registered key names, so an unknown token never becomes a key at all — it is absorbed into the preceding key’s value. Probed directly:

input:  subject=s MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/x.creds
stored: subject="s MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/x.creds"

So the rejection could not have been about the command line. It was validateConfig sweeping the whole subsystem and tripping over a different, already-stored target named ONE that carried all three keys. Only one code path in the tree writes those key names into a store: the legacy config migration. Driving the pre-fix migration on an enabled NATS target named ONE reproduces the issue’s error text character for character — including the empty nkey_seed=, which is just what migration writes when the legacy config had no NKey.

That reframes the incident. This was not “the server rejected my command.” It was: an old config was migrated once, the migration wrote three keys the validator does not accept, and the store has been failing validation at every boot since — taking every other notification target down with it, silently, because the failure is logged and swallowed. The reporter’s command merely walked into the blast radius and got handed someone else’s error.

One constant, two meanings

The declaration, as inherited (internal/event/target/nats.go, pre-fix):

const (
    NATSAddress  = "address"
    NATSSubject  = "subject"
    NATSUsername = "username"
    NATSPassword = "password"
    NATSNKeySeed = "nkey_seed"            // config key — correct shape
    // ...
    EnvNATSUsername     = "MINIO_NOTIFY_NATS_USERNAME"
    NATSUserCredentials = "MINIO_NOTIFY_NATS_USER_CREDENTIALS"  // ← in the Env block
    EnvNATSPassword     = "MINIO_NOTIFY_NATS_PASSWORD"
)

NATSUserCredentials is named like a config key, valued like an env var, and shelved with the env vars. The parser used it as both: once as the env var to look up, once as the config key to read from the stored KVS. The migration used it as a key to write. There was no "user_credentials" string anywhere in the program — the config key for creds-file auth simply did not exist, which is why the reporter, finding no documented key, resorted to passing the env var name as one.

A name that means two things will eventually be wrong in one of them. Here it was wrong in both directions at once: as a key it was unregistered garbage; as the only spelling available it taught users and the migration to write garbage.

Four surfaces, no handshake

A notify option in this codebase lives on four surfaces that must agree: the defaults (DefaultNATSKVS — what validation accepts and mc admin config get displays), the help (HelpNATS — what mc admin config documents), the parser (GetNotifyNATS — what the server actually reads), and the migration (SetNotifyNATS — what upgrades write). Nothing ties them together. Three upstream feature PRs each updated the parser and the env plumbing, and each forgot the first two surfaces:

Key Parser reads Migration writes Defaults Help Introduced
user_credentials yes (via the two-faced constant) yes (as the env-name string) no no #19139, 2024-02
nkey_seed yes yes no no #21231, 2025-04
tls_handshake_first yes yes no no #21008, 2025-04
immediate (AMQP) yes see below no no config-KV rewrite era

The AMQP row hides the quieter sibling. The AMQP migration did not skip immediate — it wrote immediate’s value under the internal key, and dropped cfg.Internal entirely:

config.KV{
    Key:   target.AmqpInternal,          // wrong key
    Value: config.FormatBool(cfg.Immediate),  // right value
},
// cfg.Internal: written nowhere

Because internal is registered, this one passes validation. The NATS gaps break a migrated config loudly enough to be found eventually; the AMQP gap corrupts it silently — a migrated broker config carries the wrong flag with a clean bill of health. One defect class, two presentations: the unregistered key fails closed, the misrouted value fails wrong.

The amplifier

None of this would deserve the word “outage” without the aggregation semantics. FetchEnabledTargets iterates the ten notify subsystems and returns (nil, err) on the first failure; its only caller logs the error and moves on, leaving the global notification target list nil; every later lookup nil-guards into an empty list. One rejected notify_nats target therefore turns off all bucket notifications — Kafka, webhook, AMQP, MQTT, the lot — with nothing but one line in the server log.

We considered changing this to per-subsystem isolation and decided not to, in this fix. Skip-the-broken-subsystem is a real behavioural change to how operators experience a bad config: today it fails loudly-in-aggregate (everything stops), and configurations that operators have already reasoned about depend on validation being all-or-nothing. Rewiring that is a compatibility decision that deserves its own change, not a rider on a registration fix — and once registration is correct, legal configs no longer trigger the cascade at all. The decision is recorded as a doc comment on FetchEnabledTargets and pinned by a characterization test, so the next person to touch it changes it on purpose or not at all.

The fix

About a hundred lines of production change, carried by nine hundred lines of tests (162ded343: 8 files, +1029/−7).

Registration. All four keys enter their default KVS and help schema, placed where an operator would look for them (user_credentials beside username, nkey_seed after token, tls_handshake_first after tls_skip_verify, immediate beside mandatory). Registration is also what makes a key visible: all four now appear in mc admin config get output where they previously did not.

The constant, split. NATSUserCredentials becomes a real config key, "user_credentials"; a new EnvNATSUserCredentials carries the env string. Every env var name involved — MINIO_NOTIFY_NATS_USER_CREDENTIALS, _NKEY_SEED, _TLS_HANDSHAKE_FIRST, MINIO_NOTIFY_AMQP_IMMEDIATE, and their _TARGET-suffixed forms — is frozen byte-for-byte: they are public interface, they worked throughout (the env route was always the workaround), and a test now pins them as raw string literals, so no rename of a Go constant can drift them silently.

Help flags, by precedent. Both new NATS values are file paths (a .creds file; an NKey seed file), so they are marked Sensitive but not Secret, mirroring cert_authority/client_cert/client_key rather than password/token. Secret would additionally redact them from mc admin config get — hiding an operator’s own configured path from them, which is why the private-key path client_key never had it either.

Migration, corrected. SetNotifyNATS now writes the real key; SetNotifyAMQP writes immediate = cfg.Immediate and internal = cfg.Internal.

If you are affected today, on a pre-fix build: the env var route works and always did, and mc admin config reset myminio notify_nats:<target> un-wedges a poisoned store at the cost of its settings. On the fixed build, poisoned stores simply load again — next section.

Living with what the old migration already wrote

Fixing the migration helps the next upgrade. It does nothing for stores the broken migration already wrote, which contain the literal key MINIO_NOTIFY_NATS_USER_CREDENTIALS — still unregistered, still fatal at every boot. Telling those operators to hand-reset their config would mean punishing them for our write.

So the load path tolerates it, narrowly. Validation accepts the legacy spelling for the NATS subsystem only — a test asserts AMQP still rejects it, so the tolerance cannot become a general escape hatch — and the parser falls back to it only when the real key is empty. Precedence is env > user_credentials > legacy key, and it holds by construction rather than by convention: the fallback result is passed as the default argument of the env lookup. All three orderings are tested. The legacy key stays out of the defaults and the help on purpose: it is tolerated, never advertised, never newly settable (kvFields sees to that).

The constant for it is a package-local literal, not an alias of EnvNATSUserCredentials — deliberately. It names bytes already on disk, so it must not follow any future rename of the env constant. The comment says so.

One trap discovered while wiring this, worth its own paragraph because it will bite someone eventually: the codebase has two CheckValidKeys — a free function and a method — and their deprecatedKeys parameters mean opposite things. The free function tolerates the listed keys (skips them); the method subtracts them from the valid set (rejects them). Refactoring this call from one form to the other would silently invert the tolerance into a ban. That asymmetry is now documented at the call site, which is the best one can do short of renaming an exported API.

The tolerance is written to be retired: the clean end state is to rewrite the legacy key into user_credentials once at load, then delete both the tolerance and the fallback. That is follow-up #2 below — and it also closes a small hole the tolerance leaves open: an unregistered key carries no Sensitive flag, so a tolerated legacy key ships its value (a path) unredacted in health-diagnostics bundles while user_credentials shows *redacted*.

Secrets in error messages

The invalid-keys error that started all this printed the rejected pairs with their values: found invalid keys (MINIO_NOTIFY_NATS_USER_CREDENTIALS=/jwt/creds/minio_notifier.creds ...). Those paths are mild. The mechanism is not: whatever value rides on a rejected key — a mistyped nkey_sed=<seed>, a bind password on a stale LDAP key — lands in the server log and in the mc client’s terminal.

Both CheckValidKeys forms now print key names only, keeping the shape and the mc admin config reset hint. The second site was one step beyond the written task scope — the method form serves LDAP, OpenID, and the policy plugin, where a rejected value can be an actual bind password — and the independent review, asked to judge that extension, said it would have demanded it: fixing one of two identical leaks is a half-fix. Repo-wide, nothing parsed values out of that string and no test asserted the old text; the change is global and intended.

There is a converse worth recording: this redaction is what stands between the next defect of this class and a credential in the logs. The follow-up below found the Postgres/MySQL migrations writing a plaintext database password under an unregistered key — on a pre-redaction build, the resulting rejection prints that password.

A guard that makes the class extinct

Registering four keys fixes four keys. The class — four surfaces, no handshake — stays open unless something ties the surfaces together mechanically. The fix therefore ships an AST-based audit test that parses parse.go and legacy.go, resolves the constants (from the target package sources, so there is no hand-maintained list to rot), and asserts, for all ten notify subsystems:

  • every key the parser reads is registered in that subsystem’s defaults;
  • every key the migration writes is registered (minus an explicit, shrink-only allowlist — next section);
  • every help entry names a registered key.

Run against the pre-fix tree, it fails on exactly the four known gaps and nothing else — which is the red proof that it measures the right thing.

The adversarial review then attacked the audit itself with a mutation harness, on the theory that a guard you cannot watch fail is a guess — the discipline the previous article argued for. Seven of its nine mutations were caught. Two were not, and both blinded the audit silently: rename the parser’s loop variable (the read-collector pattern-matched the receiver name kv), or switch a migration entry to Go’s idiomatic elided composite-literal form (the write-collector demanded a typed config.KV{...}). In both cases the collector returns an empty map, the assertion loop iterates zero keys, and the test passes vacuously. Both are refactors a maintainer would make without a second thought; one of them is what gofumpt nudges you toward.

Two hardenings closed this, each verified in both directions — with the hardening the mutation is caught; with the hardening removed (the counterfactual) the vacuous pass returns:

  • A floor assertion in the reverse direction: every registered key must be seen being read. This holds for all ten subsystems today — measured, not assumed, including the deprecated streaming_* keys read inside a nested conditional — so it costs nothing, and a blinded collector now produces one loud error per registered key (22 of them for NATS) instead of a green run.
  • A widened literal guard: typed literals that are neither config.KV nor config.KVS are skipped; untyped (elided) literals have no type to check and are now inspected rather than ignored.

Final score: ten mutations, ten caught — the harness gained one variant along the way, and the closure round swept the full suite. The audit also enforces its own allowlist in both directions — removing an entry that is still needed fails, and an entry that goes stale (the migration no longer writes that key) fails too, so the allowlist can neither grow silently nor lie about the present.

What the audit found next

The write-side check refused to go green on two subsystems that had nothing to do with issue #39. SetNotifyPostgres and SetNotifyMySQL write five keys — host, port, username, password, database — that no default KVS registers and no parser reads. These are relics of the pre-DSN configuration shape, and the migration still emits them. Driving the real helpers confirms it: a migrated Postgres or MySQL notify target is rejected on the next load with found invalid keys (host, port, username, password, database) — the same failure mode as NATS, the same every-boot persistence, the same all-notifications blast radius through the fail-fast. And password there is a plaintext database password, which is exactly the value the redaction above now keeps out of the logs.

It is deliberately not fixed in this change. The scope was locked to the NATS and AMQP gaps, and the right treatment (register the five as deprecated, or stop writing them, or both) is a judgment call that deserves its own red/green cycle. It is pinned in the audit’s knownUnregisteredWrites allowlist with a shrink-only comment, so it cannot be quietly forgotten: the day someone fixes it, the stale allowlist entry fails the test and demands its own deletion.

Open items, none in a released build as of 2026-08-04:

  1. Postgres/MySQL migration unregistered writes — major, live at every boot for anyone migrating a pre-KV config with those targets enabled.
  2. Rewrite-on-load for the legacy NATS key, then retire the F5 tolerance and fallback; also closes the health-bundle redaction gap for tolerated keys.
  3. kvFields typo folding — an unknown key name in mc admin config set is silently absorbed into the preceding key’s value instead of erroring. Pre-existing upstream wart; it protected nobody here and will corrupt someone’s subject eventually.

Review record

The change went through three gates before commit:

Gate Method Outcome
Implementer tests written first and run against the unmodified tree; the missing constant made the suite fail to compile, which is itself the red for the split; targeted reversal produced runtime reds for the rest red established for every claim
Independent adversarial reviewer detached worktree at the pre-fix commit; re-derived every red rather than trusting the report; mutation harness against the audit; probe tests for precedence edges; reproduced the reporter’s error byte-for-byte from the migration path REVISE, two demands
Closure round both demands applied; counterfactual mutation runs (with and without each hardening) prove the hardenings load-bearing; reviewer re-diffed, re-ran, re-mutated ACCEPT, 10/10

The honest accounting, in the house tradition: neither demand was a defect in the production fix. One was a lint gate (two British spellings that would have failed make test — and while fixing them, the implementer’s rewritten comment introduced a third, dialled, which the same gate caught; the demand vindicated itself in real time). The other was the audit-blindness pair above — durability of the guard, not correctness of the change. What the review did overturn was the incident’s origin story: the migration-path reproduction, the ONE target, and the kvFields absorption proof all came from the reviewer, and they change what operators should conclude — this was a boot-time outage lying in wait in stored configs, not a CLI validation quirk.

The implementer’s red phase also surfaced three bugs in its own new tests before the fix landed, recorded rather than smoothed over: a fixture that assumed stored targets are layered over defaults when config.Merge actually passes them through verbatim; a characterization test that segfaulted on a nil HTTP transport (FetchEnabledTargets dereferences it unconditionally — hostile to testing, noted, unfixed); and an early draft keyed to the very constant the fix renames, which made it pass green pre-fix — rewritten against the literal string so it pins the on-disk schema rather than the Go symbol.

Declined, and left open

Declined, deliberately:

  • Per-subsystem error isolation in FetchEnabledTargets — a compatibility decision, not a rider (above).
  • Registering or advertising the legacy key — tolerated on load, absent from defaults and help, impossible to set anew.
  • Renaming EnvNatsTLSHandshakeFirst’s odd casing — an aesthetic rename in a fork is diff noise that buys nothing.
  • Fixing the Postgres/MySQL migration here — scope-locked, pinned in the allowlist instead (above).

Left open: the three follow-ups above, and one cosmetic consequence — a store that still carries the tolerated legacy key will show it verbatim in mc admin config get until rewrite-on-load lands.

Closing

Every one of these keys worked perfectly through the environment variable, which is why three feature PRs could ship, get reviewed, get used, and never notice that the config-file half of the interface was stillborn. The parser and the schema are two descriptions of the same contract, maintained by hand, four surfaces wide — and for two and a half years nothing in the build checked that they agree.

If only one sentence survives: when two artifacts must stay identical and only convention binds them, the divergence is not a risk but a schedule — put a machine between them, then mutate the machine until you have watched it catch the drift you fear.

Follow-up status: the Postgres/MySQL discrete-field migration left open at this review point was repaired by f1ba68358 and shipped in Server 20260903. Enabled targets without a canonical DSN now explicitly prevent startup. See database notification migration; this does not close the separate tokenizer observations.

3.15 - Object Grant, Bucket Reach: When 'bucket/*' Could Rewrite the Bucket Itself

A trailing slash let an object-only IAM grant of ‘arn:aws:s3:::bucket/*’ reach bucket-level actions — including PutBucketPolicy, the one that can make a bucket public, and DeleteBucket, the issue’s own reproduction. We shipped a narrow, deny-safe fix, then chose its final size by one question: does reaching this action give the caller anything its object access does not already provide?

Status: Fixed on pgsty/silo-pkg main (3c24ad1, extended by 1f97549, scoped to its final twelve actions in d8b1fa7), released as silo-pkg v3.11.0; consumed by pgsty/minio Classification: Access-control hardening — a privilege boundary, narrowly restored Affected scope: IAM users/roles/service accounts granted only object-scoped (arn:aws:s3:::bucket/*) access, in deployments that share a cluster across tenants Tracking: upstream minio/minio issue #20449 (public since 2024, still open)

Conclusions first

  • In IAM policy matching, a bucket-level request carries an empty object name, and the matcher built its resource string as "bucket/". An object-only policy pattern — "arn:aws:s3:::bucket/*" — then matched that string, so a grant that should cover only objects also authorized bucket-level actions.
  • The dangerous one is PutBucketPolicy. A tenant holding only s3:* on bucket/* could install a bucket policy with Principal:"*" — making the bucket publicly readable or writable — or grant itself bucket-level control. Same mechanism, same class: DeleteBucket/ForceDeleteBucket (the issue’s own reproduction), PutReplicationConfiguration (exfiltration), PutBucketLifecycle (mass deletion), PutBucketVersioning, PutBucketObjectLockConfiguration, and the rest of the bucket-configuration writes.
  • The full correction is a two-directional behavior change: it tightens over-granting Allow statements and loosens over-blocking Deny statements, and it would revoke ListBucket/GetBucketLocation grants that many real deployments write as bucket/* today. That is a compatibility break, not a clean patch.
  • So we shipped a narrow fix — first six sensitive bucket-configuration writes, then, in a second pass, twelve: the bucket-level writes that hand the caller something its object access does not already give it, plus four that no handler implements. Only on Allow statements, so no Deny and no NotResource exclusion is ever weakened, with an environment-variable escape hatch. The compatibility-sensitive read/list family, CreateBucket, and three bucket writes with plausible tenant use are left unchanged, by decision.
  • Twice we claimed the change could only remove permissions, and twice an untested case said otherwise — the second time found by an independent review of a shipped release. The protected path now requires the resource to match both the bare and the historical form, which makes the property hold by construction rather than by argument.
  • The fix is red/green proven at the matcher layer and end to end through the real handlers; the object-scoped hot path is untouched.

The slash, and the empty object name

Every bucket-level S3 operation authorizes with an empty object name — checkRequestAuthType(ctx, r, policy.PutBucketPolicyAction, bucket, ""). The IAM matcher turned that into a resource string, and for the empty-object case it appended a trailing slash:

resource.WriteString(args.BucketName)
if args.ObjectName != "" {
    // "bucket/object"
} else {
    resource.WriteByte('/') // "bucket/"  <-- the defect
}

"bucket/" is matched by the wildcard pattern "bucket/*", because * matches the empty string. So a policy that grants s3:* on arn:aws:s3:::bucket/* — which reads as “anything, but only on the objects in bucket — was evaluated as granting bucket-level actions too. The bucket-policy evaluation path (for anonymous/public access) never had this slash and is the reference-correct behavior; only the IAM path was wrong, and there was exactly one place it went wrong.

This is upstream minio/minio #20449, filed in 2024. An early upstream attempt deleted the slash outright and was reverted the same day for breaking policies that relied on the old behavior. The lesson we took from that revert shaped the fix below.

What it actually enables

PutBucketPolicyHandler has a single authorization gate and nothing behind it. Once the IAM check passes, the caller may store any well-formed bucket policy for that bucket.

The concrete chain, in a multi-tenant cluster:

  1. An administrator grants tenant A the policy Allow s3:* on arn:aws:s3:::bucket-a/*, intending “A may work with the objects in bucket-a, nothing more.”
  2. Because of the slash, A may call PutBucketPolicy on bucket-a.
  3. A installs { "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::bucket-a/*" }. Every object in bucket-a is now readable by the anonymous internet. s3:* makes it world-writable. Pointing Principal at an account A controls exfiltrates the data; granting itself bucket-level actions in that policy is self-escalation.

The same object-only grant reaches other bucket-configuration writes with comparable consequences: replication to an attacker’s target, a one-day lifecycle expiry that deletes the bucket’s contents, disabling versioning, tampering with object-lock retention. None of these should be reachable from a grant scoped to objects.

This is not remotely exploitable and requires no missing credential — the caller is an authenticated principal you deliberately gave a scoped policy to. In a single-tenant deployment, that principal is your own trusted user and the practical risk is low. In a shared, multi-tenant cluster it is a real cross-tenant boundary failure.

Why a narrow fix, not the whole boundary

The obvious fix is to stop appending the slash for every bucket-level request. We did not do that, for two reasons that matter more than the one-line diff suggests.

It breaks common, benign usage. The correction does not only revoke the dangerous bucket writes — it also revokes ListBucket, GetBucketLocation, and ListBucketMultipartUploads when they were granted through bucket/*. Many deployments write exactly that and rely on it. The evidence is upstream’s own test suite: eleven STS integration tests grant s3:ListBucket on bucket/* and then assert that listing works. If the projects that wrote the server write it this way, production policies do too. A maintenance upgrade that turns those into AccessDenied is precisely the kind of surprise we refuse to ship.

It cuts both directions. The matcher builds the same resource string for Allow and Deny. So the full correction tightens over-granting Allow statements and simultaneously loosens over-blocking Deny statements: an administrator who locked a bucket with Deny s3:* on bucket/* would silently lose that protection for bucket-level actions. A clean-looking fix that moves security in two directions at once is not a maintenance patch — it is a migration.

So we narrowed the change to where it is unambiguously right and effectively free of compatibility cost:

  • Only bucket-level writes are protected. The first pass covered six sensitive configuration writes: PutBucketPolicy, DeleteBucketPolicy, PutReplicationConfiguration, PutBucketLifecycle, PutBucketVersioning, PutBucketObjectLockConfiguration. The second pass (below) extended that to twelve. Almost nobody grants these through an object-only pattern on purpose — you do not accidentally rely on an object grant being able to rewrite a bucket’s policy or delete the bucket — so revoking that path breaks essentially no one.
  • Only on Allow statements. Deny statements keep the historical resource string, so no existing Deny is ever weakened. The narrow fix only ever adds a denial.
  • The read/list family is left exactly as it was. ListBucket on bucket/* still works. That is the compatibility-sensitive part, and it waits.

The fix

The matcher keeps the trailing slash in every case except one: a bucket-level Allow statement being evaluated for a protected action, with the compatibility shim off.

resource.WriteString(args.BucketName)
if args.ObjectName != "" {
    // "bucket/object" — unchanged
} else if args.BucketName == "" {
    resource.WriteByte('/') // KMS two-phase sentinel — unchanged
} else if legacyBucketResourceMatch.Load() ||
    statement.Effect != Allow ||
    !isSensitiveBucketMutation(args.Action) {
    resource.WriteByte('/') // historical behavior for Deny / non-sensitive / shim-on
}
// else: bare "bucket" — an object-only "bucket/*" no longer authorizes it

Because args.Action is the concrete request action, a wildcard grant (s3:*) is covered too: the wildcard matches at the action step, and by the time the resource string is built the action is the specific PutBucketPolicy. A bare-bucket resource (arn:aws:s3:::bucket) and the * resource still match, so correctly scoped grants — including the built-in readwrite policy — are untouched.

The escape hatch is MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on, read once at startup. It restores the full historical behavior — both the over-grant and the over-block — for any operator who needs the old semantics while they adjust their policies.

The second pass, and the question that decided its size

The first round protected six configuration writes and stopped. Reviewing it against the original issue showed that was not enough: the action reproduced in #20449 itself — DeleteBucket — was still reachable through an object-only grant. An end-to-end test against the first-pass build confirmed it: a user holding nothing but s3:* on arn:aws:s3:::bucket/* called RemoveBucket and the bucket was gone.

Extending the set raised the real question — how far? The first instinct was “every bucket-only write except CreateBucket,” fifteen actions. That was the wrong instinct, and the reason is a detail of how the bug fires.

The bug only triggers when the statement already grants the bucket-level action. Resource matching runs after action matching, so a read-only tenant holding s3:GetObject on bucket/* never reaches DeleteBucket — the action never matched. In practice the affected principal holds s3:*, which means they already have full read, write, and delete over every object in the bucket. That reframes the severity of each candidate action, because the question is not “how dangerous is this action in the abstract” but “what does reaching it add to a position that already includes all the data?”

By that test, three groups fall out:

Protected — reaching these grants something the object access does not. PutBucketPolicy and DeleteBucketPolicy hand access to other principals, anonymous included, and can grant the caller bucket-level actions it was never given: self-escalation and public exposure. PutBucketObjectLockConfiguration and PutBucketVersioning defeat protections that exist precisely to stop a holder of write access from destroying data. PutReplicationConfiguration and PutBucketLifecycle act under server credentials and keep acting after the caller’s access is revoked. DeleteBucket and ForceDeleteBucket destroy the bucket entity and its configuration irreversibly.

Protected at zero cost. PutBucketCors, DeleteBucketCors, PutBucketQOS, and PutInventoryConfiguration have no MinIO server behavior attached today — no handler at all, or a handler that returns NotImplemented after the authorization check. Withholding them changes nothing that works, and covers them in advance if a handler is ever wired.

Deliberately not protected. PutBucketTagging, PutBucketEncryption, and PutBucketNotification are bucket-level writes, and the first draft of this pass did protect them. They came back out. None of the three gives the caller access it does not already hold — the harm is to the owner’s posture, not to the access boundary — while a tenant handed s3:* on bucket/* and told “this bucket is yours” may quite reasonably tag it, set default encryption, or wire up event notifications. Low security gain against a real compatibility cost is the wrong trade for a maintenance release. They keep the historical matching, and a test now asserts that they are unprotected, so putting any of them back is a deliberate act with a visible cost rather than an edit to a list.

That leaves twelve actions, shipped as silo-pkg v3.11.0. Two older boundaries stand unchanged: CreateBucket keeps the historical matching (it targets a bucket that does not exist yet, and provisioning flows commonly create a tenant’s bucket with that tenant’s own credentials), and the read/list family still waits for the migration-gated change.

The choice of what to break, in other words, was made by asking would an administrator ever write this on purpose — not by ranking the actions by how dangerous they sound. The first question predicts which upgrades break; the second only sets urgency.

The claim that was wrong twice

Everything above rests on one property: this change may remove permissions and must never add one. Both times we asserted it, we were asserting it about a mechanism we had reasoned through rather than tested through. Both times it was false.

The first pass withheld the slash from the NotResource match as well — and NotResource is an exclusion. An Allow s3:* NotResource bucket/* statement historically did not apply to bucket-level requests on that bucket; matching the exclusion against the bare bucket name made it stop matching, so the Allow it qualified grew, for exactly the writes being protected. Restoring the historical form for NotResource fixed that, and the second pass shipped saying the result was “provably monotone.”

An independent adversarial review of that release produced a counterexample within the hour. Withholding the slash does not merely remove a match — it changes which string patterns are matched against, and a pattern can match "mybucket" without ever having matched "mybucket/". The clean case is a fixed-width wildcard:

Allow s3:PutBucketPolicy on arn:aws:s3:::mybucke?

? matches exactly one character. Against the historical nine-character "mybucket/" it does not match, so this statement never authorized the bucket-level write. Against the new eight-character "mybucket" it matches, so the hardening granted something the buggy matcher refused. Small in reach — you have to write a length-sensitive pattern — but it is precisely the class of defect the property was supposed to exclude, shipped in a release whose notes claimed the property held.

The fix is not another special case. On the protected path the matcher now requires both forms to match: the bare bucket name and the historical "bucket/". The result is an intersection with the historical decision, so it is monotone by construction — there is no pattern it can newly satisfy, and no argument to get wrong next time. mybucket* still grants (it matched both all along); mybucket/* is still withheld; mybucke? is refused exactly as it always was. That shipped as silo-pkg v3.11.0.

Two things are worth taking from this beyond the patch itself. A correctness fix in an authorization path must never make anything newly allowed — and the only way to know is to test both directions, because the reasoning feels airtight in both cases where it wasn’t. And when a security property is load-bearing, build it out of an operation that cannot violate it rather than out of a case analysis you believe is complete.

Regression tests now pin each direction: grants narrowed, Deny untouched, NotResource exclusions untouched, fixed-width wildcards not broadened, the three unprotected writes still reachable, plus an invariant test that every protected action really is bucket-only (ResetBucketReplicationState, despite its name, is an object action and stays out). In the server they run end to end through the real handlers — client, inline session policy, and the S3 router — and every one of them fails against the release that had the bug.

What you will notice

For nearly everyone: nothing. Object access is unchanged, ListBucket via bucket/* is unchanged, and correctly written bucket policies are unchanged.

The one visible change: a request that tries to delete the bucket, or change its policy, replication, lifecycle, versioning, or object-lock configuration using credentials whose only matching grant is an object-only bucket/* pattern now returns AccessDenied. Bucket tagging, default encryption, and event notification are not affected. That is the boundary being enforced. If a deployment genuinely depends on the old behavior, set MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on and grant those actions on the bare bucket ARN (arn:aws:s3:::bucket) at your own pace.

What we deliberately left open

The general problem in #20449 — that bucket/* reaches the remaining bucket-level actions: ListBucket, GetBucketLocation, the configuration reads, CreateBucket, and the three tenant-plausible writes above — is not fixed here. Closing it fully means revoking grants that real deployments depend on, so it belongs to a future release that carries a migration path.

What that release owes operators is more than a wider action list, because no one can enumerate every deployment’s policies — which means shrinking or growing the protected set by guessing is an exercise with a hard ceiling. Three things raise it:

  • A startup policy audit. Walk the stored policies and name each one whose meaning changes, in both the grant and the deny direction. That turns an upgrade surprise into a pre-upgrade checklist, it is read-only, and it can ship before the enforcement change rather than with it.
  • A denial that explains itself. When a request is refused because only an object-scoped grant matched, say exactly that, and name the compatibility switch. A break an operator can diagnose in thirty seconds costs an order of magnitude less than a silent one.
  • A switch with a scope. MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH is all-or-nothing today: an operator who needs one action back has to reopen the self-escalation path along with it. Per-action scoping is what makes the change safe to adopt.

Recording the boundary rather than implying it: today twelve bucket-level writes are corrected. Everything else — the read/list family, CreateBucket, and bucket tagging, encryption, and notification — still honors bucket/* as a bucket-level grant, by decision, until that migration-gated change lands.

Closing

A single appended slash turned “only the objects” into “and the bucket too.” The tempting fix removes the slash everywhere and, in doing so, breaks a listing pattern half the world relies on and quietly weakens every Deny written against bucket/*. The fix we shipped removes it in exactly the place where an object-scoped grant should never have reached — the writes that can make a bucket public, and the ones that can delete it — and nowhere else. The rest is written down, waiting for a release where breaking it is something users are told to expect rather than something that happens to them.

Follow-up checked September 16: the “CORS has no handler” zero-cost argument describes early August. #71/#80 implemented per-bucket CORS, included in Server 20260903; Put/DeleteBucketCors remain protected bucket-configuration actions and must not move to the unprotected list. MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on is read during process initialization: supply it in the actual startup environment, not a later-loaded MINIO_CONFIG_ENV_FILE.

3.16 - Absent Is Not Empty: A Blank versionid and the Fail-Open It Invites

A policy that allowed deletes only when no version was named denied every one of them. The obvious one-line fix would have turned that fail-closed annoyance into a fail-open bypass on Multi-Delete. The condition value had to become the version the server actually acts on.

Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work. Classification: Policy-enforcement correctness — a fail-closed report, a fail-open trap avoided, and one narrow trim bypass closed. Not a headline CVE — see How we classify this Affected scope: Any deployment with a bucket/IAM policy using Null or StringEquals on s3:versionid; the reported break is on DeleteObject/DeleteObjects Tracking: upstream minio/minio issue #21735 (reporter iTrooz, 2026-01-10); upstream repository archived read-only since 2026-04-25

This article is now public. The primary fix shipped as recorded above; residual findings below are the investigation-time record, not a claim that publishing one fix closed the entire defect class.

Conclusions first

  • The policy engine decides Null by slice length, not by content. MinIO wrote "versionid": {""} into the condition map unconditionally, so a request that named no version still presented a length-1 slice. Null:{s3:versionid:true} — “match only when the key is absent” — could therefore never match, and Null:false always matched. The reporter’s “allow deletes only of the current object” policy denied every current-object delete (HTTP 200 envelope, per-object AccessDenied).
  • The one-line fix is a trap. “Write the key only when it is non-empty” fixes the report and simultaneously opens something worse. DeleteObjects carries each object’s version in the XML body; the condition builder reads only the query string. Drop the empty key and a body version simply vanishes from the map — read as absent, i.e. as null — so a policy meant to protect old versions would authorize deleting a specific one. Fail-closed defect, meet fail-open bypass.
  • The real fix has two parts: write the key only when a version is named, and bind it, for DeleteObject, to the effective server-resolved version (ReqInfo.VersionID) — the per-entry body value that the DeleteObjects loop already resolves — rather than to whatever the query string happened to carry.
  • A third, adjacent hole closed on the way: the builder read the version untrimmed while the object layer trims it, so a padded ?versionId=V%20 let a Deny StringEquals s3:versionid "V" be sidestepped on the read/tag/copy paths.
  • Inherited from upstream, and unfixable there. minio/minio is archived read-only, so the fix lives in the fork; this is the same getConditionValues we hardened in the condition-source work.

Absent is not empty

A condition key in a MinIO policy resolves to a lowercase name in a map[string][]string, and the engine answers Null by asking how long that slice is (silo-pkg .../policy/condition/nullfunc.go):

func (f nullFunc) evaluate(values map[string][]string) bool {
	rvalues := getValuesByKey(values, f.k)
	if f.value { // Null:true — "the key must be absent"
		return len(rvalues) == 0
	}
	return len(rvalues) != 0 // Null:false — "the key must be present"
}

The content of the strings is never read. A slice {""} has length 1. To this function, a present-but-empty value is indistinguishable from a real version ID, and both are the opposite of absent.

Now the value that fed it, as inherited (cmd/bucket-policy.go, getConditionValues):

args := map[string][]string{
	// ...
	"versionid": {vid}, // vid == "" for any request that names no version
	// ...
}

vid is the request’s ?versionId, empty on the overwhelming majority of calls. So every request, versioned or not, arrived at the engine carrying versionid: [""] — permanently length-1, permanently “present.”

The two Null directions then invert:

Request Map state Null:true (want absent) Null:false (want present)
no version named {""} (len 1) false — never matches true — always matches
?versionId=abc {"abc"} (len 1) false true
(correct behaviour) no version absent (len 0) true false

The reporter wrote the canonical “let clients delete current objects but not roll back versions” policy — Allow s3:DeleteObject with Condition {"Null": {"s3:versionid": "true"}} — and watched every version-less delete return AccessDenied. The Allow never fired because its condition tested “no version named” and the map insisted a version was always named. StringEquals cannot see the difference either ({""} and absent both fail to intersect a non-empty policy value); only Null and ForAllValues:* are sensitive to it, which is why Null is where it surfaced.

The fail-open next door

The obvious fix writes the key only when it is non-empty, and for a single DeleteObject that is completely correct: no version → absent → Null:true matches. Ship that alone, though, and Multi-Delete turns it into an authorization bypass.

DeleteObjects (POST /{bucket}?delete) does not put versions in the query. Each object carries its own optional version in the request body:

<Delete>
  <Object><Key>photo.jpg</Key><VersionId>a1b2…</VersionId></Object>
  <Object><Key>notes.txt</Key></Object>
</Delete>

The condition builder reads r.Form — the query string — and nothing merges an XML body into it. So under the naive fix, an entry that names version a1b2… in the body produces an empty query version, the key is omitted, and the engine sees absent — null. A policy written to allow only null-version deletes now matches, and the specific old version the operator meant to protect is deleted. The fail-closed nuisance from the report has become a fail-open on exactly the operation that most needs to be scoped per object.

This is the crux the reporter’s simple case hides: the condition value must be the version the server will actually act on for this object, and for Multi-Delete that value lives on a channel the condition builder never looked at.

The fix: the effective version, not a convenient one

Two mechanisms, because either alone is wrong.

1 — Represent absence honestly (cmd/bucket-policy.go). Write the key only when the request names a version, so “no version” becomes a length-0 read:

if vid != "" {
	args["versionid"] = []string{vid}
}

2 — Bind DeleteObject to the effective version (cmd/auth-handler.go, authorizeRequestWithTags). The DeleteObjects loop already resolves each entry’s body version into ReqInfo.VersionID (via checkRequestAuthTypeWithVID, cmd/bucket-handlers.go:502, a sequential loop — no shared-state race). Authorization rebinds the condition value to that server-resolved string, and deletes the key when it is empty:

conditionValuesForAuth := func(lc string, cred auth.Credentials) map[string][]string {
	values := getConditionValuesWithTags(r, lc, cred, existingTags, requestTags)
	if action == policy.DeleteObjectAction {
		// DeleteObjects carries the effective version in each XML object,
		// not in the request query. Keep authorization scoped to that entry.
		if versionID == "" {
			delete(values, "versionid")
		} else {
			values["versionid"] = []string{versionID}
		}
	}
	return values
}

An end-to-end test drives a &versionId=query-level-decoy on the DeleteObjects URL and asserts it never reaches any entry’s decision — the per-entry body value wins, the decoy is stripped.

Why DeleteObjectAction only, and not a blanket ReqInfo rebind. The tempting simplification — “always use ReqInfo.VersionID” — breaks copy. For a CopyObject, the source read is authorized as GetObject against the source’s version, which travels in the x-amz-copy-source header, and getConditionValues already extracts it there; ReqInfo.VersionID for a copy holds the destination query (usually empty). A blanket rebind would overwrite the correct copy-source version with the wrong one. Every non-delete version-aware operation (Get, Head, tagging, retention, copy-source read) carries its version in the query or the copy-source header, both of which the builder reads, and both of which are the effective version for a single object. Only Multi-Delete diverges. So the override is precisely as wide as the divergence, and no wider.

The version the server acts on is the trimmed one

One gap remained once the delete paths were correct. The builder read the version raw:

vid := r.Form.Get(xhttp.VersionID) // untrimmed

while every path that actually uses the version trims it first — newContext (cmd/utils.go:806) and getOpts (cmd/object-api-options.go:101) both strings.TrimSpace. So on non-delete version-aware actions, a padded ?versionId=V%20 presented "V " to the policy engine while the object layer read, tagged, or retained version "V". A Deny keyed on StringEquals s3:versionid "V" — “protect this exact version” — saw "V ", failed to match, and did not fire; the operation on "V" proceeded. A narrow bypass (the attacker must know the version and that a space changes nothing downstream), but a real one.

The fix trims both reads, aligning the condition value with the effective version:

vid := strings.TrimSpace(r.Form.Get(xhttp.VersionID))
// ... and the copy-source fallback likewise

DeleteObjectAction was already immune, because it uses the already-trimmed ReqInfo.VersionID. Trimming introduces no new allow: it can only make the condition value equal the version actually operated on, which tightens Deny and corrects Allow in the same direction. We proved it is load-bearing by removing only the trim and watching the padded test case go red.

What it affected

The reported break is on delete, but the underlying key is read by many actions. After the fix, every version-aware chain evaluates s3:versionid against the version the server resolves for that operation:

Call chain s3:versionid source Effective
Single DeleteObject ReqInfo.VersionID = trimmed query, via override
DeleteObjects, per entry ReqInfo.VersionID = XML-body version, via override ✓ — the fail-open closed
GetObject / HeadObject / Select query, now trimmed
Object tagging / retention / legal-hold query, now trimmed
CopyObject / CopyObjectPart source read x-amz-copy-source version, now trimmed
Anonymous 404-vs-403 probes query (read-only)
Admin / KMS / metrics / STS no version concept

A forgery route was already closed by the earlier condition-source work and is worth restating: versionid is a reserved internal key (both the versionid and canonical Versionid spellings), so a client cannot inject a second copy through the header/query merge loops. The wire parameter is spelled versionId (capital I) and lands in an inert args["versionId"] the engine never reads.

Two directions of impact, kept distinct because they have different severities:

  • Functional (the report): version-less deletes were wrongly denied. Fail-closed — an availability and usability defect, not a grant.
  • Security (the trap and the trim): the naive fix would have granted deletes of protected versions on Multi-Delete (fail-open); and the untrimmed value permitted a narrow Deny bypass on read/tag/copy. The fix closes the first before it can exist and the second where it already did.

How we classify this

We are not minting a CVE for this, and the honest reasons are worth stating.

The behaviour the reporter filed is fail-closed: MinIO denied operations the policy meant to allow. A system that is too strict leaks nothing and grants nothing; it is a correctness and usability defect, and inflating a false-deny into a vulnerability would cheapen every real entry in this chronicle, whose neighbours are authentication bypasses and path traversals.

What carries genuine security weight is not the report but its vicinity. The fail-open on Multi-Delete is real, but it is a hazard we would have introduced, not one that shipped — the value of the two-part design is that the dangerous version never existed in a build. The trim bypass did exist, but it is narrow: it requires a Deny keyed on an exact s3:versionid, an attacker who knows the version, and it only ever affected non-delete paths. We closed it because it was in reach, not because it was a headline.

So: policy-enforcement correctness, filed here because that is where we keep silent enforcement failures, with the security interest recorded plainly rather than dressed up.

The boundaries we did not cross

Two same-class residuals remain, recorded rather than silently left:

  • Multi-Delete governance-bypass follow-up. The adjacent gap was fixed by 75a6734e4 / #104 and shipped in Server 20260903, using each XML entry’s effective version for reauthorization. The Snowball observation below is a separate scope.
  • Snowball tar extraction. PutObjectExtract takes each member’s version from the tar PAX record minio.versionId after the per-file authorization, so a named version can be written that never appeared in any condition value.

Both are narrow, both are pre-existing, and both would widen the change from “fix the reported key” into “re-plumb every action’s version into ReqInfo.” We scoped to the reported surface and wrote the IOUs down here, for the same reason the previous article recorded its object-layer omission: a deliberate omission that is not written down is indistinguishable from an oversight six months later.

A related decision, declined: the sibling keys username, userid, signatureversion, and authType are still written unconditionally empty, carrying exactly the present-but-empty defect versionid just shed — Null:{aws:username:true} is always false, including for the anonymous caller it should match. Fixing them is a one-liner each and a forty-caller blast radius, and some (principaltype is never empty) do not share the bug at all. We did not bundle a broad presence sweep into a versionid fix; it is named here as the next thread to pull.

Falsification

Three experiments, in the discipline that a test you have not watched fail is not yet a test.

  • Revert both source files to HEAD. The end-to-end DeleteObjects test turned red with every version-less entry returning AccessDenied — a faithful reproduction of issue #21735 — and the unit test caught the {""} key directly (“an absent versionId was exposed to policy evaluation”). Reapply, green.
  • Remove only the TrimSpace. The padded case went red on the exact assertion — got [7f4b6b5f-…dd8 ] — proving the trim is not decoration. Restore, green.
  • The decoy. The Multi-Delete test appends &versionId=query-level-decoy to the URL and asserts it reaches no entry’s decision, which is what distinguishes “reads the query” from “reads the effective per-entry version.”

The change touched only five files (cmd/bucket-policy.go, cmd/auth-handler.go, two tests, one doc example), committed with explicit paths in a working tree that had concurrent unrelated work in it, so nothing from the neighbouring efforts was swept in.

Source and lineage

The report is upstream minio/minio#21735, opened 2026-01-10 against RELEASE.2025-09-07T16-13-09Z: a Null:{s3:versionid:true} policy denying version-less DeleteObjects. The upstream repository went archived and read-only on 2026-04-25, so there is no upstream fix to wait for and no maintainer to coordinate with — the fork is the only venue, and the record here is the resolution.

The defect is old and inherited. getConditionValues has written versionid unconditionally for as long as the key has existed; the length-based Null semantics are upstream’s, in the policy package the fork consumes via silo-pkg. This is the same function and the same lineage as the earlier condition-source hardening that stopped client input from shadowing server-derived condition values — a related read of “what a policy condition is allowed to believe about a request,” continued here into “and it must believe the version the server will actually act on.”

Closing

Absent is not empty. A map that cannot say “no version” by leaving the key out will say it by leaving the value blank, and a Null that counts length will believe a version was named on every request that named none.

If one sentence survives: a fail-closed bug is the dangerous kind to fix, because the obvious repair flips it to fail-open — so bind the condition to the value the server actually acts on, from the same channel the operation reads, not the channel that was convenient; and when you stop at the reported surface, write down the versions you left on the wrong channel, rather than trusting the next person to find them.

3.17 - Three Headers, One Lie: Making the Client Source Address Mean Something

A switch named for one header was being recommended as a defence against three. Disabling X-Forwarded-For left X-Real-IP and Forwarded answering in its place, so aws:SourceIp and every audit client address stayed forgeable by anyone who could reach the API port. The fix is an opt-in trusted-proxy boundary — and the more interesting decision was refusing to repair the old switch.

Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work. Classification: Opt-in hardening plus a documentation defect, not a vulnerability and not a regression; no CVE assigned. The underlying weakness is inherited from upstream and its default behaviour is unchanged here Affected scope: aws:SourceIp policy conditions, the audit log remotehost field, S3 event notification Host, and the client shown by mc admin trace — on any deployment whose S3 API port is reachable without passing through a header-sanitising proxy Upstream: nothing to file — minio/minio is archived. Prior art there: PR #4736 (2017, the concern raised and half-addressed), discussion #17878 (2023, maintainer marks it working as intended), PR #20977 (2025, the partial switch)

This article states plainly that an IpAddress policy condition is not enforceable on a directly-reachable MinIO deployment, and that this remains true by default after the change. That is a property of upstream MinIO as shipped, not a defect introduced by the fork, and it has never been documented anywhere. Publishing it is the point.

Conclusions first

  • MinIO reads the client’s address out of three interchangeable headers — X-Forwarded-For, X-Real-IP, RFC 7239 Forwarded — and never from the TCP connection unless all three are absent. That address becomes aws:SourceIp and the audit log’s client field, so whoever controls it controls both IP-based access control and the attribution of every logged action.
  • The one switch that existed, _MINIO_API_XFF_HEADER=off, suppresses one of those three. An attacker’s response is to send X-Real-IP instead. Our own code comment was recommending it as the mitigation.
  • “Put MinIO behind a reverse proxy” is not sufficient, for two independent reasons: on Kubernetes an Ingress and a ClusterIP Service routinely coexist so the proxy is not the only way in; and the stock nginx recipe appends to X-Forwarded-For, leaving a client-supplied entry in the left-most position — which is exactly where MinIO reads.
  • The fix is a new opt-in setting, MINIO_API_TRUSTED_PROXIES, generalising a trusted-proxy mechanism this fork already built for LDAP STS rate limiting. Set to a list, forwarded headers are believed only from those peers and chains are walked right-to-left. Set to none, nothing is believed.
  • The most consequential decision was one we reversed. The first implementation widened _MINIO_API_XFF_HEADER=off to suppress all three headers. That was the only part of the change that could alter an existing deployment’s behaviour, and it was backed out. The switch keeps its exact upstream semantics, and upstream’s TestXFFDisabled is retained unmodified as the proof.
  • Net compatibility impact: none for any deployment that does not opt in.
  • Adversarial review found four defects in the first implementation, including one that made the new setting silently ineffective for every deployment configured through an environment file.

What the address is actually used for

The value comes from a single function, handlers.GetSourceIPFromHeaders. Tracing its consumers is what turns this from a logging curiosity into a security question:

Consumer Why it matters
aws:SourceIp (cmd/bucket-policy.go) Decides IpAddress / NotIpAddress policy conditions
Audit remotehost The record used to investigate every other incident
Event notification Host Flows to downstream consumers as fact
mc admin trace client Operator’s live view of who is doing what

Two of these are security-relevant in different ways. A forged aws:SourceIp is a live access-control bypass: an IpAddress condition meant to confine a principal to an office CIDR is satisfied by asserting an address in that CIDR, and a NotIpAddress deny is evaded by asserting one outside it. A forged audit address is quieter and arguably worse — it corrupts the record retroactively, it applies even where no IP-based policy exists, and nobody notices until they need the logs.

Note also that the value is never validated as an IP address in the default path. Forwarded: for="_gazonk" is accepted and returned verbatim; upstream’s own test asserts it.

The setting that looked like a mitigation

internal/handlers/proxy.go gates exactly one header:

if enableXFFHeader {
    if fwd := r.Header.Get(xForwardedFor); fwd != "" {
        // ... left-most entry
    }
}
if addr == "" {
    if fwd := r.Header.Get(xRealIP); fwd != "" {
        addr = fwd                       // not gated
    } else if fwd := r.Header.Get(forwarded); fwd != "" {
        // ... first for= element        // not gated
    }
}

Setting _MINIO_API_XFF_HEADER=off costs an attacker one line: send X-Real-IP instead of X-Forwarded-For. Worse, disabling X-Forwarded-For moves the trust to a header the operator has not thought about, so the switch can leave a deployment in a state its owner has not modelled.

Where did it come from? Upstream PR minio/minio#20977, whose entire stated motivation is:

Customer request to disable all XFF header handling, ping me in Slack for more details.

No security rationale, no mention of X-Real-IP or Forwarded, no public discussion of why one header was gated and two were not. The narrowness is an oversight, not a considered scope. That mattered for the design, because it meant nobody had decided the other two should stay trusted — but as we will see, it did not end up justifying a change to the switch.

Upstream knew, in 2017

The most interesting thing found while writing this up is that none of it is news to upstream. The history is a small lesson in how a security decision decays.

August 2017. IpAddress / NotIpAddress condition support is added in PR #4736. During review the maintainer, @harshavardhana, raises exactly the concern this article is about: X-Forwarded-For is trivially spoofed, the left-most entry is the client’s own, and using it for a security decision would let a malicious client reach objects. The contributor accepts it and removes X-Forwarded-For support entirely, leaving only X-Real-IP, on the stated reasoning that a proxy sets it and a client cannot manipulate it. Merged five days later.

That reasoning is half right, and its unstated half is the whole problem: X-Real-IP is untamperable only if the proxy in front overwrites it. Nothing enforced that, and nothing told operators it was load-bearing.

Today. X-Forwarded-For is read first, ahead of X-Real-IP. The 2017 decision did not survive; it dissolved across later refactors of the condition-value plumbing rather than being reversed on purpose. There is no commit that says “we are re-admitting the spoofable header into policy decisions” — which is precisely how this class of decay happens.

August 2023. In discussion #17878 an operator reports that source IPs behind a load balancer are unreliable. The maintainer’s answer is unambiguous: without reliable source-IP visibility, IP-based restrictions are impractical, and the recommendation is to compartmentalise by tag or namespace instead. Marked working as intended.

So upstream’s own position — stated by a maintainer, in public — is do not rely on aws:SourceIp. That is a defensible engineering stance. What is missing is anywhere an operator would encounter it: it is not in the policy documentation, not in the condition-key reference, and not near the setting that appears to make it safe. An IpAddress condition is accepted without complaint and behaves as though it works.

That gap is the actual defect being fixed here, and it reframes the change. The allow-list is not overturning an upstream judgement; it is offering the mechanism that would make the 2017 concern answerable, to the deployments that want it. The documentation is doing the heavier lifting: writing down a contract that has been implicit since 2017 and contradicted by its own switch since 2025.

Why “put it behind a proxy” is not the answer

This is the standard advice, and it fails in two common, independent ways.

The proxy is not the only way in

On Kubernetes an Ingress and a ClusterIP Service routinely coexist. The Ingress sanitises headers; the Service does not, and any pod in the cluster can reach it. The security boundary is assumed to be the Ingress but is actually the pod network. The same shape appears in Pigsty deployments, where a load balancer fronts MinIO while the service ports remain reachable on the internal network.

The canonical nginx recipe preserves attacker input

The near-universal snippet is:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

$proxy_add_x_forwarded_for expands to $http_x_forwarded_for, $remote_addr — it appends to whatever the client sent. A client sending X-Forwarded-For: 1.2.3.4 causes nginx to forward 1.2.3.4, <real client>. MinIO takes the left-most element, which is the attacker’s.

So a correctly-proxied, hardened, no-direct-access deployment is still forgeable, because left-most parsing and append-style proxies are mutually incompatible. Only proxy_set_header X-Forwarded-For $remote_addr; (overwrite) is safe under the default mode, and that is not what operators copy from the documentation.

This is the finding that rules out a documentation-only fix. Deployment discipline cannot close it; the chain has to be read from the other end, which requires code.

The design

Rather than inventing a mechanism, we generalised one this fork already has. MINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIES — added during the LDAP STS throttling work — already implements CIDR allow-list parsing, catch-all rejection, and a right-to-left chain walk, scoped to rate-limit bucketing. The parser moved to internal/config and both paths now share it.

One setting selects the mode:

Mode MINIO_API_TRUSTED_PROXIES Source address
Untrusted (default) unset unchanged from today
Trust nobody none always the TCP peer
Allow-listed addresses and CIDR blocks forwarded headers, only from listed peers

Under the allow-listed mode, X-Forwarded-For and Forwarded are read right-to-left, stepping over entries that name a listed proxy, and the first remaining address wins. Each proxy appends the peer it actually saw, so an entry the client injected sits to the left of the one its proxy wrote and the walk stops before reaching it. Appending proxies become safe.

GetSourceScheme is deliberately untouched. It feeds the Location URL in S3 responses rather than a policy decision, and suppressing it would hand http:// URLs to every deployment terminating TLS at a proxy.

Trade-offs

Whether to change the default

Changing the default to distrust forwarded headers would make every existing reverse-proxy deployment’s aws:SourceIp and audit addresses become the proxy’s address overnight. Policies could fail closed; audit continuity would break.

Not changing it leaves directly-reachable deployments exposed.

Decision: do not change it. But “do not change” is not the same as “stay silent”. The consequence is now written into the code comment and the operator documentation in as many words: under the default mode, an IpAddress condition is not access control and the audit address is not evidence. Making the cost visible so operators can choose is better than making a choice for them that detonates during a release window.

Whether to widen the existing switch

This is the decision we got wrong first and reversed, and it is the part of the story most worth recording.

The original brief asked that when an operator explicitly disables forwarded-header trust, clients must not be able to forge through an equivalent header. The obvious reading is “fix _MINIO_API_XFF_HEADER=off so it covers all three”, and that is what the first implementation did.

The case for widening was decent. Upstream’s PR title says “disable all X-Forwarded-For header handling”; its described scope is audit logs and IP-based access control, both of which are about not believing client-claimed addresses; the switch is undocumented, so its audience is small; and the failure direction is safe, since the fallback is the real TCP peer rather than an attacker-controlled value.

The case against turned out to be decisive. Widening it changes behaviour for a real population: operators whose proxy appends to X-Forwarded-For (polluted) but overwrites X-Real-IP (clean) may have discovered off as a way to get the correct address. Upstream’s TestXFFDisabled asserts exactly that behaviour — with the switch off and both headers present, X-Real-IP wins — so the behaviour is not merely incidental, it is pinned by a test. Those operators would have seen audit addresses silently flip from the real client to their proxy, and IpAddress conditions potentially start denying.

The reversal came from separating the goal from the mechanism. The goal was “a complete, enforceable way to turn this off exists”. Nothing required that the existing variable be the thing that provides it. Expressing it as MINIO_API_TRUSTED_PROXIES=none achieves the identical guarantee with zero effect on anyone who has not opted in.

Decision: leave _MINIO_API_XFF_HEADER exactly as upstream defined it. It gates X-Forwarded-For only, within whichever trust mode is in force. Upstream’s TestXFFDisabled is retained unmodified and still passes. The documentation now says plainly what the switch is not: it is a parsing switch, not a trust boundary, and a client refused one header simply sends another.

A secondary benefit: this collapses two interacting variables into one policy setting, so there is no longer a precedence rule ("off outranks the allow-list") for operators to learn and for us to get wrong.

Environment variable or config subsystem

Registering trusted_proxies as an api subsystem key would give mc admin config visibility, help text, and hot reload, matching how sts_trusted_proxies is done.

It would also introduce a window. Config subsystems load after the object layer initialises, so between process start and config application the trust policy would be empty — which under a “list is empty means trust everyone” reading is fail-open. A security boundary must not have a fail-open window. Separately, a trust boundary that can be changed at runtime is not obviously desirable.

Decision: environment variable. The cost is discoverability, and one bug described below.

Loopback is always trusted as a peer

The FTP and SFTP front-ends connect to the S3 layer over 127.0.0.1 and declare their session’s client with X-Forwarded-For (cmd/sftp-server-driver.go). An allow-list that does not exempt loopback attributes every FTP and SFTP request to the server itself.

The cost is that any process on the same host can forge. That is acceptable: an attacker who can open connections from localhost already has code execution on the host, and the threat model is lost well before this point. The FTP/SFTP regression, by contrast, would be certain and would affect everyone using those front-ends.

Decision: exempt loopback as a peer. Adversarial review then caught that the first implementation also treated loopback as a skippable chain entry, which is unnecessary for the FTP/SFTP case and actively harmful — see below. The two are now separate checks.

X-Forwarded-For versus X-Real-IP: genuinely unresolvable

Under the allow-listed mode, which header wins when both are present?

  • A proxy that authors only X-Real-IP and relays the client’s X-Forwarded-For (some nginx configurations) → preferring X-Forwarded-For takes the forged value.
  • A proxy that authors only X-Forwarded-For and relays the client’s X-Real-IP (AWS ALB) → preferring X-Real-IP takes the forged value.

Both are common, and the server cannot tell which situation it is in from the request. This is not an undecided question; it is undecidable without the operator telling us what their proxy authors.

Decision: prefer X-Forwarded-For. It is the only one of the two that carries a chain that can be checked against the allow-list, and this path decides access control rather than rate-limit bucketing, so the value that can be validated should win. It also keeps header precedence identical to the default mode, so switching modes does not silently change precedence as well.

This deliberately diverges from getSTSLDAPTrustedProxySourceIP, which prefers X-Real-IP. Two contradicting implementations of the same question in one codebase is a hazard in itself, so both sites now carry a comment naming the divergence and its reason, so that nobody “unifies” them without re-deciding. The operator documentation states the mitigation for both directions: strip whichever header your proxy does not author.

A broad allow-list is worse than no allow-list

This is the most counter-intuitive property and the one most likely to bite.

Entries on the list are skipped during the chain walk. So MINIO_API_TRUSTED_PROXIES=10.0.0.0/8, configured because the load balancer is 10.0.0.1, makes every client inside 10/8 skippable as well. A client at 10.5.5.5 sending X-Forwarded-For: 8.8.8.8 produces a chain of 8.8.8.8, 10.5.5.5; the walk steps over 10.5.5.5 as a “trusted hop” and returns 8.8.8.8.

A broad list therefore does not merely trust more peers — it lets those peers forge. nginx’s set_real_ip_from has the same property with real_ip_recursive.

There is no algorithmic fix: the list is doing double duty as “who may forward” and “whose address may be discarded”, and separating them would mean two lists to keep in sync. Decision: keep one list, and make the constraint prominent — a callout block in the operator documentation, and the rule stated in the code comment where the walk happens. Only /0 is rejected as a catch-all, and the documentation says explicitly that this is a guardrail rather than a proof, since 0.0.0.0/1,128.0.0.0/1 covers the same ground.

Multi-node forwarding

MinIO forwards requests between nodes for bucket-DNS routing, listing continuation, heal-by-token, batch jobs and pool decommissioning. The receiving node’s TCP peer is the forwarding node, not the client.

Mode Receiving node resolves
Default the client
none the forwarding node
Allow-list without node addresses the forwarding node
Allow-list with node addresses the client

This is not an obscure path: a ListObjectsV2 continuation token carries the node index, so any client can cause its own request to be forwarded. Under none, that request is then evaluated with aws:SourceIp set to an internal node address — which an IpAddress condition allowing internal ranges would treat as a pass.

Decision: document it, and steer multi-node clusters to the allow-list. none cannot be corrected for this case, because it believes nothing by definition. Automatically seeding the cluster’s own addresses was considered and rejected: it needs DNS resolution at startup and re-resolution as node addresses change, which is more machinery and more failure modes than the explicit configuration it replaces.

Compatibility

The change was deliberately structured so that risk is not spread evenly across it. Every piece is either opt-in or dead code in the default configuration.

Change Who is affected Risk
MINIO_API_TRUSTED_PROXIES allow-list only those who set it none
Forwarder sanitises X-Real-IP / Forwarded code path does not execute in default mode none
Startup failure on a malformed value only those who set it, incorrectly none
Policy re-read after environment-file load same result when nothing is set none
LDAP parser extracted for sharing nobody — pure code motion, verified identical none
_MINIO_API_XFF_HEADER semantics nobody — reverted none
_MINIO_API_XFF_HEADER read timing nobody — upstream timing kept deliberately none

Evidence for the default path. unverifiedSourceIP is a verbatim copy of the original function body, including its quirks: the ", " separator, the fall-through when the left-most element is empty, and the acceptance of non-IP values such as _gazonk. An independent review verified behavioural parity against HEAD over 21 cases — empty X-Forwarded-For, a bare comma, a leading ", ", "," versus ", " separators, " , ", IPv4-mapped addresses, bracketed IPv6, non-IP junk, and all three Forwarded forms. Upstream’s TestGetSourceIP and TestXFFDisabled are both retained unmodified and pass.

A subtlety self-review caught. The new setting is read after MINIO_CONFIG_ENV_FILE is loaded, which is what makes it work in packaged deployments. The obvious tidiness move is to read _MINIO_API_XFF_HEADER in the same place — and that would have been a behaviour change, because upstream reads it at package initialisation, before environment files exist. An operator who wrote it into an environment file has it silently ignored today; picking it up would make an already-deployed setting suddenly start working, flipping their source addresses from the left-most X-Forwarded-For entry to X-Real-IP. The old switch therefore keeps upstream’s read timing along with upstream’s semantics, and a test pins that so nobody tidies it later. The quirk is documented instead: set it in the process environment if you want it honoured.

The defensive code that was not defending anything. Sharing the parser initially came with two extras: allow-list entries written in IPv4-mapped form were rewritten to the IPv4 prefix they denote, and the address being matched was unmapped and de-zoned. Both looked like corrections — an entry written ::ffff:192.168.1.10 is otherwise accepted and then matches nothing, which is a silent failure worth removing.

They were removed anyway, and the reason is worth recording. Both call paths reduce the address through net.ParseIP(...).String() before matching, and that already collapses ::ffff:10.0.0.1 to 10.0.0.1; a dual-stack listener reports an IPv4 peer in plain form regardless. So neither extra could be reached by a real request. Their only observable effect was on what the shared function meant for the LDAP STS allow-list that had been using it first — 18 differences that a test could see by calling the function directly and no deployment could.

Worse, one of them manufactured the fail-open described below: rewriting ::ffff:0:0/96 turned a /96 into 0.0.0.0/0. Deleting the rewrite removes the bug’s cause rather than ordering around it. What remains is pure code motion, verified identical to the previous implementation across every combination of 37 allow-list values and 21 peer addresses — zero parse differences, zero match differences. The wart it declined to fix (a mapped-form entry matches nothing) is the pre-existing behaviour, fails closed, and is now stated in the function’s own comment so the next person does not re-derive the same tempting fix.

The one residual risk worth naming. The default mode’s code path did change: there is now a switch and a function call in front of the original body. If that plumbing were wrong it would affect everyone, not just opt-in users. The parity testing above is why we believe it is not, but “verified equivalent over 21 cases” is a different claim from “provably identical”, and the honest version is the former.

What adversarial review found

An independent agent was tasked with breaking the first implementation. It found four real defects, all since fixed and covered by regression tests.

A silent fail-open, and the worst of the four. The trust policy was read in the package’s init(). But loadEnvVarsFromFiles() calls os.Setenv for everything in MINIO_CONFIG_ENV_FILE long afterwards — which is how MinIO is configured in essentially every packaged deployment. An operator putting MINIO_API_TRUSTED_PROXIES in /etc/default/minio would have got the historical trust-any-peer mode, with no error reported, and a malformed value would have been silently ignored rather than fatal. The policy is now applied in serverHandleEnvVars, which runs after the file load and before any listener.

Loopback skipped as a chain entry. Described above: the peer exemption was being reused as a hop exemption, which is a needless instance of the broad-list problem. Now two separate checks.

Two fail-closed correctness bugs. A zoned IPv6 peer (fe80::1%eth0) canonicalised to nothing, because net.ParseIP rejects zones — so such a peer could never be a trusted proxy. And an allow-list entry written in IPv4-mapped form (::ffff:192.168.1.10) was accepted at startup and then matched nothing at all, since netip.Prefix.Contains is false across differing bit widths.

Two further defects were found while writing the documentation rather than the code, which is its own small lesson:

Repeated header lines. Header.Get returns only the first header line. HAProxy’s option forwardfor adds a second X-Forwarded-For line rather than extending the first, so Get would hand back the client’s line and put the forged value right back where the right-to-left walk exists to avoid it. Now flattened across all lines with Header.Values.

The internal forwarder relayed client claims. internal/handlers/forwarder.go set X-Real-IP only when absent, so a client’s value passed between nodes unchanged. Under an allow-list that includes the cluster’s own nodes — the configuration we recommend — a client could thereby borrow a peer node’s authority. The forwarder now drops X-Real-IP and Forwarded when the incoming peer is not entitled to have set them. X-Forwarded-For needs no such handling, because Go’s ReverseProxy appends the true peer and the receiving node’s walk reaches that entry first.

A second round, after the rework

Reworking the setting warranted a second adversarial pass, which was worth running: differential testing found zero behavioural differences against HEAD across 4,745,520 source-IP resolutions and 345,600 forwarder rewrites, but it also found three more ways to fail open — one of them introduced by the first round’s own fix.

A catch-all smuggled in as an IPv4-mapped prefix. MINIO_API_TRUSTED_PROXIES=::ffff:0:0/96 is a /96 as written, so it passed the catch-all check; the rewrite that unmapped IPv4-mapped entries then turned it into 0.0.0.0/0, trusting every peer. The first fix moved the breadth check to the far side of the rewrite. The eventual fix deleted the rewrite, once it became clear it was unreachable by any real request — which removes the cause instead of guarding its output.

This is the one most worth dwelling on. The fail-open was manufactured by a fix for an unrelated fail-closed bug, and the fix for the fail-open was a reordering that left the manufacturing step in place. Two rounds of correction, both defensible, neither addressing the fact that the code should not have been there. Hardening changes deserve the same adversarial treatment as the code they harden, and “is this reachable at all?” belongs near the front of that treatment.

A deliberate value naming nobody. MINIO_API_TRUSTED_PROXIES="," parsed to an empty list and fell back to the permissive default. An empty unset variable must mean “default”, but a value the operator actually typed which names no proxy is a mistake, and answering it with trust-everyone is the one behaviour they cannot have wanted. It is now a startup error. Whitespace-only remains equivalent to unset, since that is what an empty shell variable expands to.

A remote value that could not be read. MinIO supports env:// indirection, where a variable’s value is fetched from a remote webhook. env.Get discards the error from that fetch and returns the empty string — which this code would have read as “unset”, reinstating trust-any-peer at exactly the moment the operator’s intent could not be determined. The setting is now read through env.LookupEnv so the error is surfaced and startup stops. This is a general hazard for any security-relevant setting read via env.Get, and worth remembering beyond this change.

A third pass, attacking from angles the first two shared

Both earlier passes attacked the resolver as a unit. Two things that neither could see:

Nothing had tested that the trust policy reaches a decision. Every test to that point checked what the resolver returned, and the one test at the policy layer only asserted that aws:SourceIp equalled the resolver’s output — in the default mode. So a version where the resolver was correct but the policy engine read something else would have passed everything. There is now a test that drives a forged X-Forwarded-For through getConditionValues into a real IpAddress evaluation under each mode: believed by default, ignored under none, ignored from an unlisted peer, and still honoured from the listed proxy. It passes, but it should have existed before the change was called done.

The allow-listed mode had a resource amplification the default mode does not. The chain was flattened into a slice before being walked, so a client behind a trusted proxy could turn the 1 MiB header allowance into roughly 33 MB of slice headers per request — around thirty-fold — plus a million-iteration walk. The default path never had this, because it uses strings.Index on the raw header. The walk now scans backwards over the header text in place, allocating nothing, and stops after 100 hops; real chains are a handful, the answer sits at the right-hand end, and running out of budget yields no address, which falls back to the peer. A test pins the zero-allocation property, because it is the kind of thing an innocent-looking refactor would undo.

Also corrected in this pass: the deployment contract was stated too narrowly. “The proxy must overwrite whichever headers it sets” misses the case that actually bites — a proxy that correctly authors only X-Real-IP or only Forwarded still relays the client’s X-Forwarded-For, and that is the header read first. The general rule, now stated as such, is to strip every source-address header the proxy does not itself write.

One reported finding was reviewed and not treated as a defect: the catch-all guard rejects /0 and nothing else, so 0.0.0.0/1,128.0.0.0/1 covers the same ground and is accepted. Tightening it would mean rejecting broad-but-not-/0 prefixes in the parser now shared with the LDAP allow-list, newly failing configurations that are valid today, to defend against a value no deployment realistically holds. The documentation states plainly that the check is a guardrail rather than a proof, and the callout about naming proxies instead of subnets is where the real defence lives.

Recommendations

By topology:

  • Proxy you control, API port genuinely unreachable otherwise. Set nothing. But verify whether your proxy overwrites or appends: if the config says $proxy_add_x_forwarded_for, you are forgeable today. Switch to $remote_addr, or adopt the allow-list.
  • Direct exposure, no proxy. MINIO_API_TRUSTED_PROXIES=none.
  • Kubernetes or Pigsty, Ingress plus reachable Service. The allow-list, containing the proxy addresses and the MinIO node addresses. This is the configuration that makes an IpAddress condition mean anything.
  • Any multi-node cluster. The allow-list with node addresses, not none.

Two rules apply to every allow-list deployment. Name proxies, not the subnet they occupy. And strip at the edge every source-address header your proxy does not itself write — listing a peer means believing all three headers from it, they are consulted in a fixed order, and a header your proxy leaves alone is entirely the client’s. A proxy that correctly authors only X-Real-IP, or only Forwarded, still relays the client’s X-Forwarded-For, which is read first.

What was not done

  • Automatic seeding of cluster node addresses. Feasible via EndpointServerPools, but it needs DNS resolution and re-resolution on address changes. Explicit configuration was judged the smaller risk.
  • Registration as an api config key. See the fail-open window above. Revisitable if observability turns out to matter more than the startup guarantee.
  • ExistingObjectTag/*. This adjacent condition-source defect was subsequently fixed by 2f55347f7 and shipped in 20260804: only loaded stored tags supply this condition, not client request tags. This does not add tag loading to every S3 operation.

Verdict on severity

Default behaviour matches upstream, and MinIO has never documented aws:SourceIp as trustworthy on a directly-reachable deployment, so “the default is unsafe” is closer to a documentation defect than a vulnerability. One thing is squarely a defect, though: an operator set an explicit security switch and it did not do what its name and its only public description implied, and it failed silently. That is worth a record, scoped to the incompleteness of upstream’s _MINIO_API_XFF_HEADER rather than to anything the fork introduced.

minio/minio is archived, so there is no upstream to coordinate with — the same position as CVE-2026-42600.

3.18 - Sorted Is Not Increasing: How One Duplicate Part Number Doubled an Object

A 5 MiB part uploaded once, assembled twice, returned as a 10 MiB object with HTTP 200. The predicate was strict; the verb was not. Two refactors over a decade preserved the defect faithfully.

Release status (verified 2026-09-13): the primary fix in this article is included in 20260804 and later releases. The investigation below retains its original test boundaries; see the component matrix for current unreleased work. Classification: Data correctness, not a vulnerability — see Why this is not a CVE Affected scope: All backends, any authenticated S3 client, on its own upload Tracking: pgsty/silo #49

This article is now public. The primary fix shipped as recorded above; residual findings below are the investigation-time record, not a claim that publishing one fix closed the entire defect class.

Conclusions first

  • sort.SliceIsSorted with a < predicate does not test strict increase. It tests the absence of an inversion. Equal neighbours contain no inversion, so [1,1] was accepted.
  • Upload one 5 MiB part, complete with [1,1], and the server returns HTTP 200 and a 10 MiB object. The upload is then consumed: a corrected retry gets NoSuchUpload. The client cannot recover.
  • Inherited from upstream, and old. The check has had this shape since 2016-08. Two refactors — 2017 and 2023 — rewrote it faithfully, because each preserved the predicate, and the predicate was never the problem.
  • The fix is one loop at the handler layer. The object layer is left undefended by decision, and that IOU is written down here rather than left implicit.
  • Three independent reviews found no defect in the fix. What they found was a comment that misstated why a neighbouring guard exists — and, through that comment, an unrelated node-level panic.

The verb, not the predicate

The code, as inherited:

if !sort.SliceIsSorted(complMultipartUpload.Parts, func(i, j int) bool {
	return complMultipartUpload.Parts[i].PartNumber < complMultipartUpload.Parts[j].PartNumber
}) {
	writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidPartOrder), r.URL)
	return
}

It reads as “reject unless the part numbers strictly increase.” It does not do that. IsSorted evaluates the predicate in the reversed direction only — for each neighbouring pair it asks less(i, i-1), i.e. “is this element smaller than the one before it,” and reports unsorted the moment one such inversion appears. For a pair of equal elements that question is false. No inversion, therefore sorted.

The consequence is worth stating precisely, because it is what makes the misuse survive review: no strict predicate can make IsSorted reject duplicates. The only spelling that works is the non-strict one — passing <= as the less function, so that equal neighbours register as an inversion. To ask for strictly increasing you would have to write the operator that reads as not strict. Every reviewer who checked that the predicate said < was checking the right character in the wrong function.

Our replacement drops IsSorted rather than trying to spell it correctly:

for i := 1; i < len(complMultipartUpload.Parts); i++ {
	if complMultipartUpload.Parts[i-1].PartNumber >= complMultipartUpload.Parts[i].PartNumber {
		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidPartOrder), r.URL)
		return
	}
}

The rejection-set delta is exactly one class: lists containing an adjacent equal pair. Everything previously rejected is still rejected; everything previously accepted, except duplicates, is still accepted. Non-adjacent duplicates come along for free — a strictly increasing sequence is globally distinct, so [1,2,1] and [1,3,2,3] are caught by the inversion they are forced to contain.

Two refactors preserved it faithfully

The archaeology is the most transferable part of this incident.

When Shape What changed
2016-08 sort.IsSorted(CompletedParts(parts)) already present at server: Move all the top level files into cmd folder (#2490)
2017-11 same call, Less moved onto an exported type Add public data-types for easier external loading (#5170)
2023-04 sort.SliceIsSorted(parts, func(i,j) bool { … < … }) simplify sort.Sort by using sort.Slice (#17066)

Both refactors were correct as refactors: they preserved behaviour exactly, which is what a refactor is supposed to do. The 2023 commit was a repository-wide cleanup with no bearing on multipart semantics at all. It carried the < across unchanged, and the < was never wrong — CompletedParts.Less needs < to be a valid sort.Interface.

The defect lived in the relationship between the predicate and the function it was handed to, and a refactor that moves the predicate cannot see that relationship. A decade, three shapes, one behaviour: an ordering check that answers a question adjacent to the one it appears to answer.

What it actually did

Measured against both erasure backends, through the real signed HTTP handler:

Uploaded Completion list Response Resulting object ETag suffix
one 5 MiB part [1,1] 200 OK 10,485,760 bytes -2
two 5 MiB parts [1,2,2] 200 OK 15,728,640 bytes -3
one 5 MiB part at 10000 [10000,10000] 200 OK 10,485,760 bytes -2

The ETag suffix is the part count the server believes it assembled. There is no internal disagreement to detect: the metadata, the size, and the ETag are all mutually consistent and all wrong. The object simply is not what was uploaded.

Two properties make this worse than a bad error code.

The upload is consumed. Assembly runs to completion and cleans up the multipart upload, so the corrected retry returns NoSuchUpload. A client that notices the wrong size cannot fix it by resending the right list; it has to start the whole upload over, if it still has the data.

It is reachable by accident. No adversary is required. Any client that appends a part to its completion list twice — a plausible bug in a resumable-upload wrapper, a retry path, or a list built by concatenation — silently gets a doubled part instead of a 400.

Why this is not a CVE

It belongs in this chronicle because it is a silent server-side correctness failure, and this is where we keep those. It is not a vulnerability, and we are not going to inflate it into one.

The request must carry the caller’s own credentials, address the caller’s own upload, and the damaged object is the caller’s own. There is no cross-tenant effect, no privilege change, no disclosure, and no path to another account’s data. What breaks is the guarantee that a completed multipart object equals the bytes you uploaded — serious, but a correctness guarantee, not an access-control boundary.

The entries around this one in this chronicle are authentication bypasses and path traversals. Filing this beside them under the same label would make every label in the table mean less.

The boundary decision, and what it costs

The object layer has no duplicate defence at all. erasureObjects.CompleteMultipartUpload sizes its output slice to the request (cmd/erasure-multipart.go:1249) and then resolves each requested part number against current metadata (:1255). The same number resolves twice, writes two identical ObjectPartInfo entries, and adds its size twice. AddObjectPart does deduplicate by part number, but it deduplicates the metadata slice, not the request. The 5 MiB minimum-size rule cannot help either, because the duplicated part is individually legal.

We fixed the handler and left that alone. The reasoning:

  • It is the only entrance where a client-controlled list exists. The other four callers — batch, restore, decommission, rebalance — build their lists server-side from oi.Parts or 1..n, and are strictly increasing by construction.
  • The required output is an S3 error code, which is an API-layer concern. The object layer’s error vocabulary maps to a different code, so intercepting lower would hand clients a less accurate diagnosis.
  • Minimality. This fork ships narrow fixes, and a change in the assembly loop is not narrow.

The cost, recorded rather than implied: the uniqueness invariant now has exactly one enforcement point, and nothing enforces the enforcement. No compiler error and no test failure will greet the person who adds a fifth caller to the object layer; they will get a silently corrupted object. That is the same species of IOU the previous article recorded about getVolDir, and it is written down for the same reason: an unrecorded deliberate omission is indistinguishable from an oversight six months later.

What we deliberately did not add

Part numbers need not start at 1 and need not be consecutive. [1,3], [5,9] and [3] are all legal S3, and all still complete successfully.

This matters more than it sounds. “Also require the list to start at part 1” is a one-line addition that looks like tightening, would pass a casual review, and would break legal clients — anything that abandons a part after a failed upload and completes with what it has. The temptation is real precisely because the fix next door is about validating the same list.

So two test cases exist for no purpose other than to make that change fail. We verified they do their job by injecting the constraint and confirming that exactly those two cases went red and nothing else did. A guard rail nobody has fired once is a guess.

The one behaviour change we did not intend

A 14-input differential against the pre-fix build turned up exactly one behavioural change beyond duplicate rejection: [0,0] and [-1,-1] — lists that are both duplicated and out of range — moved from InvalidPart to InvalidPartOrder. Both are HTTP 400.

We accepted it, on the principle that a format error should outrank a state error: an ordering violation is decidable without reading any storage, while part existence is not. It also only affects requests that were going to fail regardless, so no client that previously succeeded can now fail.

On S3 fidelity itself we are making a documented inference, not a measurement. AWS defines InvalidPartOrder as the parts list not being in ascending order, and documents that part numbers may be non-consecutive; duplicates are not ascending. We did not verify this against a live AWS endpoint, and two independent reviewers reached the same conclusion by the same documentary route, which is agreement, not evidence.

Falsification, and a comment that was wrong

Two mutation experiments, in the discipline the previous article argued for — a test you have never watched fail is not yet a test.

Inject “must start at part 1.” Exactly the two gap cases went red; the four lists starting at 1 stayed green. The guard rail is targeted, not incidental.

Delete the neighbouring len(Parts) == 0 guard. The expected result was that an empty completion would produce some wrong-but-orderly error. The actual result was that the process panicked: the empty list reaches a storage decorator that indexes element zero of the part-path slice without a length check, on a goroutine that no recover can reach. The S3 face is masked by that one guard line, which has been there since 2022 and is not documented as load-bearing. It is tracked separately as an unfixed node-level defect, which is why this article is held.

And the part worth publishing at our own expense: the comment we wrote about that guard was wrong. It said dropping the length check would let an empty completion succeed — the opposite direction of the truth, and specifically the direction that understates danger. It was caught in review and corrected before the commit. A comment that misstates why a check exists is exactly how the check gets deleted three years later by someone tidying up.

Three acceptances, zero blocking findings

The change went through three independent gates before commit:

Gate Method Outcome
Author revert the fix, watch the test go red at the measured 10 MiB, reapply, watch it go green red/green established
Independent reviewer rebuilt the red state in its own detached worktree rather than trusting the report; 14-input differential no blocking finding
External model, different vendor read-only sandbox, independent derivation of the rejection-set argument and of the AWS reading conditional accept; the condition was that it could not compile in its own sandbox

Stated plainly, because the honest version is less flattering than the table: none of the three found a defect in the fix. What review produced was the corrected comment and, through the mutation it prompted, the discovery of the unrelated panic. That is still a good return, but it is not the same as catching a bug in the patch, and the record should say which one happened.

The rebuilt-red-state detail is the one worth copying. A reviewer who reruns the author’s tests is checking the author’s arithmetic; a reviewer who reconstructs the broken state independently is checking the author’s claim.

Declined, and left open

Declined, deliberately:

  • Two test additions — completing onto a pre-existing object, and giving each part distinct content so ordering is verified rather than just total size. Both are real improvements. Both were declined under a standing rule that this fork ships correctness and security fixes rather than test expansion, and the core invariant is already pinned by “the rejected request left no object and the upload still works.”
  • XML root element name is not validated. A document with the wrong root but correct <Part> children is accepted. This is not a bypass — the same list still goes through the same check — it is pre-existing, and tightening it risks breaking real SDKs over namespace handling. Recorded, not fixed.

Left open, none of it in a released build as of 2026-08-03:

  • Object-layer defence in depth for part uniqueness (see above).
  • The empty-list panic in the storage decorator, tracked as a node-level defect.
  • XML strictness, including <PartNumber>abc</PartNumber> returning 500 where 400 MalformedXML is correct.

The concurrent checksum work on completion (#46, #48, #50) was fenced off from this change entirely and shares no code with it.

Closing

The predicate was strict. The verb was not. A decade of review read the predicate — including the two commits that rewrote the line.

If only one sentence survives: check what the function does with the comparison, not just what the comparison says, and when you decide to leave the layer underneath undefended, write it down where the next person will trip over it, rather than trusting that they will re-derive your reasoning.

4 - Design Records

Product requirements, implementation decisions, compatibility reviews and release-readiness records.

Design records capture the reasoning behind SILO maintenance decisions: the problem being solved, the compatibility boundary, rejected alternatives, implementation requirements, and the evidence required before release.

Read the date and status first: a proposal describes work awaiting implementation or evidence; merged on main establishes source inclusion; released links an actual tag or release record. Historical decisions and deferrals do not automatically describe current behavior. Prefer fixed commits, PRs, stable tests and release records; distinguish local checks, remote CI, artifacts and production deployment.

4.1 - Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence

#77 is a reproduced site-replication correctness defect. A receiver replaces source time with arrival time and may then reject a genuinely newer deletion. Some configuration types stop exporting their timestamp after deletion, preventing heal from recovering a delete missed during an outage. Adding a DELETE branch alone cannot solve both problems.

Merge status (2026-09-12): the repair and research archive were merged into main through PR #180, commit 48ec10312. #77 is closed. All nine pre-merge CI checks passed.
Review boundary: the plan went through four Claude Code Opus 5 Max review rounds, passing the final two. Full implementation review, remediation review, and focused final acceptance returned GO_WITH_NONBLOCKING_NOTES in all three rounds. Final blockers are zero; the requested full cmd and final lint checks have now passed.
Applicability: this page describes the repair now included in main. Check the specific version of downloads and running installations; full deletion recovery still requires every node to be upgraded and deletion export to be enabled consistently.

Existing work and scope

The earlier release notes, security hardening record, and Server compatibility page already document the #77 deletion limitation. They do not provide a complete record of its state model, alternatives, or verification boundaries. This page supplies that reasoning.

The source repository archive retains the original reproduction, plan versions, final reports and invocation identities for all seven review rounds, finding dispositions, executed test logs, source/binary hashes, and a rerunnable two-site driver. Raw model reasoning streams, binaries, and temporary lab volumes are excluded. Original artifacts and copies with normalized workstation paths and document links have separate hashes.

Existing work What it repaired What it does not establish
#91 Per-site configuration counts, Policy/Quota reporting, malformed-field isolation Accurate counts do not prove convergence of values and source times
#103 Serialized writes to the whole .metadata.bin record A lock cannot validate an ordering decision made before acquiring it
#76, #78 Object Lock’s replication payload and existing-bucket adoption protection They do not replace a consistent source-state comparison for six types
CORS replication repair A separate per-bucket CORS deletion register and replication trust boundary Its semantics cannot be applied blindly to every metadata type

This repair covers Policy, Tags, SSE, Quota, Versioning, and Object Lock. Lifecycle/expiry has its own merged-payload time semantics; CORS keeps its separate mechanism. Notification, IAM, object replication, MRF, resync, and public counters are not rewritten. Object replication reliability has a separate record.

The supported release target is the maintained silo, silo-console, mc, and silo-pkg stack. Compatibility with unmodified upstream MinIO/MC remains best effort and does not require downgrading maintained components or recreating dependency forks.

What the reproduction established

The original regression reproduced on both ErasureSD and 16-disk Erasure ObjectLayers. A peer PUT originated at a past time T, but the receiving disk stored current arrival time. A subsequent DELETE carrying T+1 minute was rejected as older. RPC success alone therefore cannot establish correct final state.

Configuration Original PUT preserves source time Established empty-event behavior Original timestamp export without payload
Policy No Delete Yes
Tags No Delete No
SSE No Delete No
Quota No Delete; zero-value JSON has separate semantics No
Versioning No No operation Not used as deletion
Object Lock No No operation Not used as deletion

Three further entry-point defects matter: an old bulk event can overwrite a newer field; a request that checks state before queuing for the lock can act on an obsolete decision; and remote Tag heal omits UpdatedAt. Each requires a repair at the actual entry point, beyond changing source selection in heal.

The facts a field needs

Reuse the existing payload, field UpdatedAt, and bucket Created. No disk format, SDK field, or persisted deployment ID is added.

State Conditions Eligible source
Unknown or invalid Unknown creation time, invalid payload, or field time before creation No; missing information cannot mean deletion
Empty baseline Empty payload at zero time or Created No
Live baseline Valid nonempty payload at zero time or Created Yes, for historical configuration initialization
Real update Valid nonempty payload later than Created Yes
Real deletion Empty payload for a deletable field, later than Created Yes; this timestamped empty value is a tombstone

Empty Versioning and Object Lock events remain no-ops. Sharing a helper must not give these types deletion semantics. Zero field times use Created as a comparison baseline; this does not turn historical emptiness into a new delete.

Ordering first gives real states priority over baselines, then compares source time between real states. At the same time, a deletion wins over a live value. Conflicting live values at the same rank use a stable content key, with the lexicographically greater key winning. For timestamped peer events and heal, an identical effective state causes neither a save nor a notification. A local write allocates a new monotonic revision even when the visible payload is unchanged. Live baselines also use content-key ordering; a later creation default cannot outrank a real change.

This is deterministic conflict resolution. It does not mean a lexicographically greater configuration better expresses business intent. Operators must still choose and resubmit the intended value after conflicting concurrent changes.

Comparison must match persistence

Policy sets are map-backed, so ordinary JSON encoding can depend on iteration order. The Server sorts the complete JSON tree of an already validated policy, including Statement, Action/NotAction, Resource/NotResource, Principal, and Condition. Sid and numeric precision are retained. This adds no syntax unsupported by the existing parser.

That parser accepts NotAction and NotResource, but the original structure’s required Action/Resource encoding can fail on the corresponding empty sets. Explicit field encoding addresses this, and Policy GET/admin export use it too. The purpose is to keep an accepted policy readable, beyond making its comparison key deterministic. GET/export now sort Statement arrays that formerly followed stored order, and sort set arrays whose map-backed order could vary between calls. Authorization is unchanged; upgrading does not rewrite every stored policy.

Quota keys use the existing parsed representation encoded as JSON. {}, JSON null, and valid zero-quota documents remain live documents, not implicit deletion events. Older senders converted a zero-quota PUT to deletion for peers while keeping a local document. mcli quota clear sends a zero quota; upgraded peers retain that live zero document, with no capacity enforcement either way. An empty Policy instead follows the established peer deletion interpretation: a valid empty-policy PUT succeeds, and a subsequent GET returns the existing NotFound response.

XML configurations use the bytes of a valid document. There is no general XML canonicalization layer. Versioning needs one exception: apply the existing Object Lock constraint before comparing the effective document that will actually be persisted. Otherwise comparison can accept a value that Save rewrites, causing heal to send it again next time.

Lock the decision as well as the save

The six fields share one .metadata.bin record. Loading raw state, validation, comparison, modification, and saving must all happen under the existing metadata.lock. Comparing outside the lock still permits stale decisions; separate field locks would allow whole-record read/modify/write operations to overwrite one another.

A local write allocates its time inside the lock:

max(current UTC time, Created + 1ns, current field time + 1ns)

This lets a local correction advance beyond an already stored future field time. Timestamped peer events retain their original time. Identical or older timestamped peer states return without a write; this is not a no-op guarantee for local PUT/Delete.

Bulk processing handles only explicitly supplied fields, validates them, and saves at most once. Omission preserves a field; explicit null follows its type’s semantics. An invalid field cannot leave half the bulk update persisted. Import assigns a common time for the selected six-type fields under the final per-bucket commit lock. Disk state and outbound events use that commit’s final snapshot, not a later read combined with an earlier time. A normalized empty Policy uses the existing dedicated deletion event so bulk omitempty cannot lose it.

The save helper returns its normalized snapshot. Public Update/Delete signatures stay unchanged, while other metadata retains its existing processing and notification behavior.

Adoption must not manufacture a deletion

Adoption can change Created. Moving it earlier while retaining an empty field’s old default timestamp makes an empty baseline appear to be a real deletion. Moving it later can invalidate historical initial values.

The narrow repair, under the existing adoption lock, rebases only these six fields whose previous time was zero or equal to old Created. Actual update and deletion times remain unchanged. An empty payload alone is not proof of a default, and existing-bucket configuration protection is not redesigned. Genuinely different bucket generations still require operator intervention.

One rule across the real entry points

Entry point Required behavior
Local S3/Admin writes Monotonic time under the lock; use the committed snapshot for outbound events where needed
Typed peer events Compare and persist original source time under one lock; retain existing types and legacy Object Lock payload compatibility
Bulk/import Explicit field presence and atomic save; allocate import time at final commit
Initial synchronization Preserve historical live baselines; include real deletions after opt-in
Local/remote heal Use the same comparison for selection and apply, with complete source times; unknown IDs and failed peers do not block healthy targets
Status export Value and time belong to one record; gate newly exposed deletion times during rollout

Initial synchronization retains its existing five-type send path. Versioning is still initialized through MakeBucketHook and aligned through heal. No extra initialization path is added merely to make the table symmetric.

Heal filters valid candidates before selecting the maximum, instead of seeding from the first map entry and only then filtering defaults. Public mismatch counters cannot be the sole gate: equal content with different source times still needs synchronization. Conversely, equal effective states need no further write or RPC.

Why rollout needs a default-off switch

The startup setting MINIO_SITE_REPLICATION_METADATA_TOMBSTONES defaults to off. It controls visibility of newly exposed deletion information and does not detect remote capability.

Behavior off on
Source-time ordering and atomic apply Active Active
Ordinary deletion events Still replicated Still replicated
Existing Policy deletion-time export Preserved Preserved
Real deletion-time export for absent Tags/SSE/Quota Hidden Exported
Additional real deletions in initial synchronization Existing behavior Include all four deletable types

Old implementations cannot safely consume all newly exposed deletion information. For example, old Quota heal can clear the payload while retaining an already parsed cache value. An instruction to upgrade does not itself isolate this rolling-upgrade window, so the default stays off.

Upgrade every node at every participating site to a build containing the repair, ensure consistent settings within each site, and drain old requests. Then set on consistently and restart. Before a downgrade, first set off and restart every fixed node, then roll back the software. The old software’s original defects return with it.

While off, hidden Tags/SSE/Quota tombstones can cause repeated stale heal RPCs that a fixed receiver rejects. A subsequent heal with zero RPCs is expected only when complete state is visible and stable.

Retained and rejected alternatives

Decision Reason
Retain one internal six-field helper The same source-time defect was reproduced across all six; shared ordering prevents entry-point drift while preserving type-specific deletion behavior
Keep the whole-bucket lock, persistence fields, and heal interval They provide atomicity, durable deletion state, and missed-event recovery without another coordination service
Do more than replace UTCNow with source time That alone leaves stale lock-external decisions, invisible deletes, equal-time conflicts, and initialization gaps
Do more than export tombstone times Old receivers and Quota cache handling remain unsafe without controlling the rollout window
Do not use deployment ID as a tie-breaker or add an HLC/schema Existing time and content keys suffice for the bounded contract; cross-site causal ordering is not claimed
Do not reject all zero-time typed events Old Tag heal really omits time; preserve inexpensive protocol compatibility with an explicit limitation
Do not impose one deletion rule on all metadata That would delete Versioning/Object Lock or violate separate Lifecycle/CORS rules

The initial implementation adds 729 and removes 692 production Go lines, a net increase of 37, chiefly replacing duplicated apply and heal branches. Line count does not establish minimality. Necessity must connect each mechanism to a concrete failure; sufficiency must cover every real entry point; minimality asks which failure returns if a mechanism is removed.

Validation and its limits

The environment is local go1.27.1 darwin/arm64. Four groups of pre-implementation audit tests failed on the unfixed baseline. The resulting regression suite passes on both ObjectLayers; its main entry points are in cmd/site-replication-metadata{,-heal,-gate}_test.go. The baseline audit log retains the original failures alongside the passing existing tests.

Validation Observation
Source time, four deletions, duplicate/reordered events, queuing before the lock, different-field writes Regressions and targeted race checks pass
Both equal-time arrival orders, deletion priority, Policy keys and negative-set GET, bulk/import Boundary regressions and supplemental race checks pass
Full cmd package and internal/S3 Select race Full cmd passes on final production code fcbb93e89 (492.776 seconds); internal/S3 Select race passed at 62cf066ff
Build, vet, lint, generated files, compatibility checks Final production build/vet pass; lint reports zero issues at 461e9a721. Generated-file and compatibility checks passed at 62cf066ff. Optional typos is unavailable and skipped according to the Makefile
Linux/Darwin/Windows × amd64/arm64 Six cross-compiles pass at 62cf066ff; this is not runtime acceptance on six platforms
Two real site processes, four data directories per site Initial synchronization preserves Created for six historical live configurations; normal 30-second heal restores consistency after dropping a real PUT’s outbound RPC and injecting reordered events
Missed deletions and restart Four deletion states persist across source-process restart and converge after reconnection
Quiescence and diagnostics Two separate 65-second observations contain no metadata RPC across two normal heal cycles; repeated exceptional events deduplicate by bucket/field/reason
Fixed and pinned old implementation together Tags PUT/DELETE smoke passes with all switches off; this does not prove complete mixed-version correctness
Review repairs: creation-time recovery, policy status key, heal diagnostics Real ObjectLayer creation-time and legacy-order policy tests fail with old production code overlaid and pass after repair. Full cmd, internal packages, S3 Select race, lint, generated files, branding checks, and six cross-compiles were rerun at 62cf066ff
Final cleanup regressions Targeted diagnostic, initial-sync, physical-time boundary, adoption, and CORS race tests pass at fcbb93e89; related race tests pass again after test-style changes in 461e9a721
Two-site acceptance repeated on the final binary The same run passes on binaries built from clean 62cf066ff and clean fcbb93e89; final 461e9a721 only changes test style and has identical production code

Local evidence contains baseline failures, test logs, a rerunnable two-site driver, metadata snapshots, and source/binary SHA-256 identities. The first two-site binary reports 01aaef2b5 + dirty; its actual Go build-info and SHA-256 have now been recorded. After review repairs and cleanup, runs were repeated on binaries built from clean 62cf066ff and fcbb93e89. Actual --version, Go build metadata, and SHA-256 identities are retained with the baseline binary built from 5c5765816. Final 461e9a721 differs from fcbb93e89 only in test formatting and equivalent conditional syntax; that diff is recorded separately.

The table above records local tests and isolated-process observations. Remote integration evidence is separate: all nine checks on PR #180 passed, comprising six Go CI jobs, DCO, vulnerability analysis, and release-pipeline validation. The merged main tree was verified to match the PR merge tree that passed these checks. This is not proof of real Linux multi-node cluster behavior, official release artifacts, or production deployment.

Adversarial review record

The plan used actual Claude Code claude-opus-5 --effort max for four rounds. The first two drove corrections to state comparison, committed snapshots, historical baselines, and import boundaries. The last two returned GO_WITH_NONBLOCKING_NOTES with zero pre-implementation blockers. Plan approval is not proof that implementation is correct.

The separate implementation review pinned 1ee64a8d8 and used the same model and effort to inspect the complete diff, production call chains, formal tests, and runtime evidence independently, challenging sufficiency, minimality, rollout safety, and evidence identity. Its verdict was GO_WITH_NONBLOCKING_NOTES: no unconditional blocker, one conditional blocker, and nine further findings. It confirmed the core convergence mechanism and found no counterexample to ordering, deletion, or duplicate suppression within the declared contract. Three findings were real defects the change had introduced and were repaired in 62cf066ff. The subsequent fcbb93e89 closes the diagnostic gap when no valid source exists and adds an initial-sync regression.

Finding Assessment Disposition
F1: a bucket with no recorded creation time can no longer write any of the six configurations Confirmed regression, conditionally blocking Repaired. GetBucketInfo returns the physical probe unchanged for a metadata-free request, as ListBuckets already did; initial synchronization recovers the time and passes it to the bucket creation hook
F2: replication status compares statement order while heal compares the canonical key Confirmed; a permanent false mismatch that heal can never resolve Repaired. Status compares the same key heal compares; per-site presence counting is unchanged
F3: one log key for four heal conditions, at error level for a normal transient Confirmed Repaired. Each reason keeps its own key at warning level. Empty baselines and missing buckets stay quiet; invalid existing state is still diagnosed when no valid source exists
F4: three user-visible semantic changes not written down Partly valid The original reviewed README already documented empty Policy and zero Quota at its end; the first review missed them. The addition covers Policy GET/export ordering and negative-set behavior
F5: whether the Policy encoder has unnecessary callers The second review corrected the first assessment Comparison and status keys must agree. GET/export/peer need the encoder to read or replicate negative-set policies that can already be persisted. PUT/import normalization is optional for comparison, but removing it adds branches and representation differences, so it stays
F6: an orphaned helper and a stale ordering comment Confirmed nits Comment corrected. The unused isBucketMetadataEqual and the obsolete test of that helper have been removed
F7: with the switch off, missed Tags/SSE/Quota deletions do not converge A correct reading of the plan’s trade-off No change; stated in rollout and in the limits below
F8: evidence gaps - a stubbed recovery test, no legacy-order policy case, unverifiable binary identity Confirmed Recovery now runs on the real ObjectLayer; a permuted legacy policy case was added; the two-site acceptance was repeated on a binary built from the clean final tree, with identities recorded
F9: bucket generation conflicts Declared out of scope, and not a regression No change; see the limits below
F10: adoption moving Created later than a real field time A coverage gap, not a defect The case now fixes both sides: preserve historical time, exclude earlier-generation state as a source, and accept a valid adopted-generation input as a target

A stub is worth calling out separately. The original recovery test injected an object layer whose creation probe returned the expected time, so it passed against code that could never behave that way in production. The replacement stamps the bucket directory on every local drive and drives the real object layer, and it fails on the unrepaired code.

The second review pinned 62cf066ff, again using actual claude-opus-5 --effort max. It returned GO_WITH_NONBLOCKING_NOTES with zero conditional or unconditional blockers. It retraced production paths, checked the author’s F1/F2 reproduction results from formal tests with old production code overlaid, and revised the first review’s assessment of the Policy encoder. The reviewer did not execute the tests.

Follow-up finding Final disposition
NB-1: physical Created is an approximation Retain the generation boundary and document directory-mtime limits. A real-drive test fixes the behavior: an earlier peer event is skipped and a local correction succeeds. One source timestamp cannot lower bucket identity
NB-2 and NB-8: silence without a source; lost source time in recovery-error logs Repaired. Invalid existing states remain visible and recovery failures retain the event time. No source means no RPC; empty baselines stay quiet
NB-3: outage logs multiply by bucket and field Accept and document the existing bucket/field/reason granularity; a site-wide log aggregation framework is outside this correctness repair
NB-4: draft logs are not product-failure evidence Mark findings-before.log and findings-after-1.log as superseded fixture failures. Formal tests with old production code overlaid reproduce the physical-time, policy-order, initial-sync, and diagnostic defects separately
NB-5 and NB-6: unused helper; incomplete recovery-path description Remove the helper and both tests that only exercised it. Document initial sync’s persisted recovery and retain real CORS-path tests
NB-7: adoption covered only the source side Add the target side: valid adopted-generation input replaces invalid earlier-generation state

Diagnostic regressions now use the existing logger target to check warning level, distinct reasons, repeated calls, and zero RPCs. The initial-sync test drives a real source ObjectLayer and the complete outbound sequence; its peer only acknowledges RPCs. That proves outbound content. The real two-process experiment supplies a separate level of evidence, and the two must not be conflated.

The third focused acceptance pinned final production code fcbb93e89 and again returned GO_WITH_NONBLOCKING_NOTES, with zero blockers. It also inspected the test-only style diff in 461e9a721 and confirmed equivalent semantics. Full cmd and lint were still running when the reviewer read their logs; both subsequently exited with code 0. Test formatting caused lint to fail at fcbb93e89 itself; corrected 461e9a721 is the delivery baseline that satisfies the required checks.

Three nonblocking improvements remain outside this repair: heal diagnostics can show a zero source time after decode/parse failure; the initial-sync unit test does not execute the local peer branch and therefore does not prove that recovered Created is persisted locally (that production path was reviewed); and strict system-log capture could need isolation if concurrent background logging is introduced. No additional production accessor or test hook was added for these points. A counterfactual failure proves the assertion actually reached, such as empty-baseline noise. Warning-level and per-reason deduplication assertions pass after repair; that is not a claim that each was separately demonstrated failing on old code.

Operational boundaries that remain

  1. Historical timestamp pollution is not reconstructable. Arrival-time replacement and old untimestamped events have lost source facts. Installing the repair cannot recover their true historical order. Inspect every site and resubmit the intended configuration or deletion at the authoritative site.
  2. Zero-time typed events remain compatible. They use monotonic local time and emit legacy-zero; the existing zero-time constraint for bulk remains. These events are outside the timestamped-source convergence guarantee.
  3. Bucket identity conflicts are not merged automatically. Resolve generation differences first. Events before target Created are not applied; unknown creation time is recovered only from a real physical bucket. Unknown or missing buckets are not written. An existing bucket whose physical creation time is still zero produces an InternalError (500) and aborts initial synchronization; a missing bucket instead retains NoSuchBucket (404). An invalid existing field can also block a typed write with InternalError; heal may replace it using a valid peer source. The recovered value is a physical approximation from bucket-directory modification time. It can change with top-level entries, differ across drives, and be later than actual creation. Older source events are still skipped; one event is insufficient to lower the bucket identity. A successful configuration write or initial site sync records the recovered value. Until then, status reports the unknown time and periodic healing skips the bucket in both directions.
  4. A physical clock is not a causal clock. Local correction can advance beyond an already known future field time, but cannot infer the business intent of all concurrent writes.
  5. Diagnostics are bounded; success does not mean applied. legacy-zero, before-created, indeterminate, unreachable, and peer-error reuse LogOnceIf with stable keys and error text, details in attributes, and existing hourly cleanup. Each reason keeps its own key, so one condition cannot deduplicate another away. Empty baselines and peers that do not yet have the bucket stay quiet. Invalid existing states emit indeterminate even when no valid source can be selected, without causing an RPC. Normal duplicates and older events remain quiet. This is a per-bucket/field/reason bound: both unreachable-peer warnings and indeterminate-state warnings can grow with bucket and populated-field count, not a fixed site-wide limit.
  6. Code, integration, and release require separate acceptance. Issue state, Server version, images, packages, published documentation, and production settings each need their own evidence. The main-branch merge recorded here does not establish that release artifacts or production installations have been upgraded.

Related records: tags · metadata · HTTP · audit

4.2 - An Unsigned Header Is Not Part of the Request

This record describes the unsigned-header coverage repair committed to SILO as 123325430 and merged through PR #173, tracked as SN-2026-011. It was reported by Oren Yomtov against a released build and reproduced locally on both signature paths.

Status on 2026-09-11: the original repair is pushed and merged through PR #173. The follow-up signing and payload-verification fixes described below are also merged through PR #177, with all eight PR checks passing. Source validation and published releases are separate: the currently published September 3 Server release does not contain these fixes.
Scope: SigV4 header coverage, consistent policy inputs and body-checksum verification. S3 field names, object and bucket metadata formats, replication protocols, encryption formats and client commands are unchanged.
Security property for ordinary signed and presigned SigV4: unsigned client-supplied x-amz-* operation headers cannot change an authorized request; policy evaluation and body verification use the effective signed inputs.

Too Long; Didn’t Read (TL;DR)

A presigned PUT URL can sign only the host header. SILO confirmed that each header named in the signed-headers list had arrived, but it never walked the headers that actually arrived, so an x-amz-* header outside that list was accepted and used. cmd/api-router.go routes any PUT carrying x-amz-copy-source to CopyObjectHandler on that header alone. Together these turned a write grant for one object into a server-side copy that reads any object the signing key can reach, executed as the signer — a confused deputy. The Authorization-header path behaved the same way when the header was left out of SignedHeaders.

The repair states one invariant:

On ordinary signed and presigned SigV4 paths, received x-amz-* headers
must be signed, except for the separately bound X-Amz-Content-Sha256.

This matches AWS S3, which refuses the same request with AccessDenied (“There were headers present in the request which were not signed”). The affected signing code was inherited from upstream minio/minio. The released SILO baseline predates this repair; source provenance alone does not establish the status of every upstream build or other fork.

Failure: the coverage gap

extractSignedHeaders in cmd/signature-v4-utils.go iterates the signed-headers list and, for each name, pulls the value from the request (or the query string). It proves that every promised header is present. It never asks the opposite question — is every x-amz-* header that arrived actually in the list?

The one place that walked the arriving headers, checkMetaHeaders, matched only the X-Amz-Meta- prefix and was called only from the presigned path (doesPresignedSignatureMatch). The Authorization-header verifier (doesSignatureMatch) called nothing equivalent. So an unsigned x-amz-copy-source — or any other operation-shaping x-amz-* header — sailed through on both paths:

presigned PUT (SignedHeaders=host)  ->  add unsigned  x-amz-copy-source: /src/secret
  -> router sees x-amz-copy-source  -> CopyObjectHandler
  -> copy runs as the signer, reading a bucket the URL never named

Reproduced locally on RELEASE-style builds: the control PUT returns 200 with an empty body; the same URL plus the one unsigned header returns 200 with a CopyObjectResult whose ETag is the md5 of the victim object, and the destination reads back the victim’s bytes. Where the destination bucket already allows anonymous GetObject, the copied private bytes are then readable with no credentials at all.

Provenance

The gap is inherited from upstream MinIO; SILO did not introduce it. The SigV4 verifier in cmd/signature-v4-utils.go and the Authorization-header path doesSignatureMatch in cmd/signature-v4.go are original MinIO code dating to 2016, and the header-driven CopyObject dispatch in cmd/api-router.go traces to 2019. The only routine that ever walked the arriving headers, checkMetaHeaders, was added upstream on 2023-07-27 in minio/minio#17737 (535f97ba6). Upstream therefore recognized the class — an unsigned header must match the signed set — but scoped the check to the X-Amz-Meta- prefix and to the presigned path, leaving x-amz-copy-source and the whole Authorization-header path uncovered. The inherited verifier predates the SILO fork.

Before the unsigned-header repair, SILO’s change to cmd/signature-v4-utils.go was the one-line dependency-path migration in 9b11dc946, moving the policy import to pgsty/silo-pkg/v3. The vulnerable verification behavior came from upstream. The original repair (123325430) and the follow-ups in PR #177 change that boundary. The vulnerable code predates SILO’s fork baseline, the upstream 2025-12-03 maintenance-mode commit from which the first SILO release was cut.

This record establishes the repair in the maintained SILO source graph. It does not claim that SILO is the only implementation with a fix, or establish the current maintenance status of other projects.

The repair

checkMetaHeaders becomes checkUnsignedHeaders, is broadened from the X-Amz-Meta- prefix to all of X-Amz-, and is called on both the presigned and Authorization-header paths. A header that is not covered by the signed set is refused with ErrUnsignedHeaders before any handler logic runs.

Four decisions shaped the exact boundary. Each had a plausible alternative that was rejected for a concrete reason.

Membership, not value equality

The inherited check compared signedHeadersMap.Get(k) == val[0]. For a header absent from the signed map, Get returns the empty string, so a header whose first value is empty compared equal and passed. A multi-value header such as X-Amz-Copy-Source: ["", "/src/secret"] could therefore smuggle an unsigned copy-source past a value-equality check. The repair tests membership in the signed set instead. A signed header’s value is already bound by the signature, so value equality was never the property that mattered; presence in the list is.

Exempt X-Amz-Content-Sha256

X-Amz-Content-Sha256 can be omitted from SignedHeaders because the effective payload hash is bound separately in the canonical request. For presigned requests, the query value takes precedence, with a header fallback when the query value is absent. An explicit UNSIGNED-PAYLOAD remains valid. PR #177 aligns the policy condition with this effective value while preserving header-presence semantics, and checks header-only presigned body hashes in the generic authentication path as well as upload paths. The exception does not permit policy evaluation or body verification to use a different value.

Derive signature age from the signed date

The original repair exempted an internal x-amz-signature-age scratch header written after verification. That was too late for PUT and UploadPart authorization, which runs before signature verification. PR #177 instead derives s3:signatureAge directly from the signed X-Amz-Date, and removes the scratch header, its constant and its exemption. A forged date fails signature verification; an unsigned client header under the old name is rejected. Verification remains idempotent without mutating request headers.

Inject X-Amz-Tagging after authentication

PutObjectTaggingHandler derives an X-Amz-Tagging header from the request body so that policy conditions can read it, and it previously did so before authenticateRequest. With the broadened check, that server-synthesized header — which the client never signs — would be refused as unsigned. The injection now happens after signature verification and before authorization, which still has it for policy conditions. Rejected alternative: blanket-exempt X-Amz-Tagging the way content-sha256 is exempt. That would let a client set object tags through an unsigned header on any signed or presigned write, reopening a smaller version of the same class of bug.

Scope across signature modes

  • Authorization-header (signed) and presigned SigV4: both now enforced. These are the reachable paths.
  • Streaming SigV4: the seed verifier does not call checkUnsignedHeaders. The copy handlers reject streaming authentication through their ordinary authentication dispatch, but that does not establish complete header coverage for streaming PUT/UploadPart paths. No dedicated streaming-copy rejection test is claimed here. A universal coverage guarantee requires separate implementation and regression evidence.
  • SigV2: unaffected. V2 canonicalization folds the x-amz-* headers into the string-to-sign by construction, so an added x-amz-* header changes the computed signature and is rejected as a signature mismatch.

Status code: 400 versus 403

AWS returns 403 Forbidden for an unsigned header; SILO returns 400 AccessDenied (ErrUnsignedHeaders), inherited from upstream. The attack is refused either way, and the error Code string is identical; only the HTTP status differs. Raising it to 403 is a one-line change to cmd/api-errors.go that also shifts the pre-existing meta-header rejection. It is left as a deliberate, reversible election rather than folded silently into a security fix, because it is a behavior change for the existing unsigned-meta-header path and is not required to close the vulnerability.

Tests

Several existing tests built a signed request and then set x-amz-copy-source, x-amz-copy-source-range, or x-amz-metadata-directive after signing — that is, they depended on the very behavior this fix removes. They now re-sign with signRequestV4 after setting those headers, as required for these operation-shaping headers by the repaired verifier. signRequestV4 excludes the Authorization header from its own signed set, so re-signing is safe. Current coverage includes checkUnsignedHeaders unit cases for empty first values, the payload-hash exception and rejection of the obsolete unsigned signature-age header and TestPresignedVerifyIdempotent, which verifies the same presigned request twice.

Evidence

  • A built server reproduced the confused deputy on both the presigned and Authorization-header paths, then refused both after the fix while the control PUT, a real minio-go CopyObject, PutObject with user metadata and tags, and body-based PutObjectTagging all continued to work.
  • go test ./cmd/ passes on the fix tree; gofmt, gofumpt, and vet are clean.
  • Adversarial review (round one) independently surfaced three defects in the first draft — non-idempotent verification via the scratch header, the empty-first-value bypass, and over-rejection of an unsigned x-amz-content-sha256 — each of which is addressed above and confirmed by re-running the reviewer’s own adversarial test suite against the final tree.
  • Adversarial review (round two, against the committed fix) found no regression and confirmed the re-signed tests keep their original intent: an invalid access key still returns InvalidAccessKeyId, and a wrong SSE-C key still returns 403 after the signature validates. It surfaced three adjacent, pre-existing gaps that also fail on the parent commit and are out of this change’s scope; they are recorded under follow-ups below.

Compatibility and operations

  • Ordinary clients: no request change. Conforming ordinary signers include the nonexempt x-amz-* headers they send. Custom signers must verify this contract; streaming modes have the separate boundary above.
  • Unsigned x-amz-* headers: now refused with AccessDenied, as on AWS. A client that added such a header without signing it was already outside the SigV4 contract.
  • Rolling upgrade: wire and storage formats are unchanged. Upgraded nodes enforce the boundary; nodes still running an older build remain exposed until upgraded, so behavior can differ by node during the rolling window.
  • Rollback: data written by the fixed version stays readable by the previous version, but rollback reopens the confused deputy.

Residual risks and follow-ups

  • Release delivery: a source fix and a public engineering record do not establish that a published binary or image contains the fix. Verify the selected release and artifact separately.
  • CVE: the reporter requested one; the finding carries the stable fork-local SN-2026-011 identifier until a CVE is assigned.
  • Status code election: the 400-versus-403 choice above is open.
  • Adjacent signing fixes: PR #177 addresses repeated copy-source ambiguity, signature-age authorization ordering, and the effective payload-hash policy value. It also closes the separately reproduced header-only presigned body-checksum gap. The regression set covers signed and presigned requests, policy enforcement before upload verification, and real HTTP bucket-policy tampering. These follow-ups are distinct from the original SN-2026-011 finding; their merge and release status is recorded above.
  • The general question: this repair covers x-amz-* request headers. Any future control that lets request syntax select an operation must answer the same question this one did — is this value covered by the signature before it is allowed to mean anything? The repeated-header gap above is the same question in a different guise: the value the signature binds and the value the handler consumes must be the one and the same.

Conclusion

The signature is the request. Everything an x-amz-* header claims is a claim until the signature covers it:

Confirming that the promised headers arrived is not the same as confirming that the arrived headers were promised. Refuse any unsigned x-amz-* header before the handler runs, on the ordinary signed and presigned paths covered by this repair. Streaming coverage remains a separate boundary.

The ledger separately tracks the payload-verification repair as SN-2026-012. The signing change alone does not change storage formats, but the same main candidate also contains IAM changes requiring coordinated upgrade. Do not use this record as approval for a rolling upgrade of that entire candidate.

4.3 - Durable IAM Revocations

Source status, 2026-09-16: #191 and #192 are merged into main, but are absent from published Server 20260903. They change persistent IAM state and require a coordinated upgrade. Follow the upgrade and recovery runbook and SN-2026-013.

SILO retains the version of a deleted IAM record so an offline site cannot restore an older identity or grant when it reconnects. This covers built-in users, service accounts, groups, policy documents, and policy mappings in their actual user/STS-parent/group namespaces. It also revokes the deleted built-in user’s older service accounts, STS credentials and group grants across deliberate same-name recreation.

Ordering and persistence

Deletion records occupy the original IAM configuration paths. They contain the originating timestamp, Deleted, and, where required, RevokedBefore; identity deletion records contain no secret key or session token. Normal IAM listings and authorization hide deleted records. Object storage and etcd both serialize each path’s version comparison and write with a distributed lock. The source timestamp is persisted without replacing it with the receiving node’s clock.

Older events cannot overwrite a newer revision. At an identical timestamp, a deletion wins over a live record. A local deliberate recreation receives a version newer than the stored deletion. An older user/group deletion arriving after recreation retains its revocation boundary while preserving the newer live record. Receiving an already-applied tombstone does not rewrite or advance it. These rules also apply after cold loading persistent IAM state.

The user or group revision is the commit point of deletion. Cleanup of mappings and children follows that commit; cleanup failure cannot undo it. The API returns the cleanup error and still notifies sibling nodes to reload the committed state. Such an error does not mean the identity is still active. Retry the intended revocation only while IAM writes and same-name recreation are paused. When the API returns a committed-cleanup error, the admin handler does not send its immediate cross-site hook; cross-site propagation relies on the normal deletion-healing retry. Sibling deletion notifications reload current shared state, so a delayed notification cannot delete a subsequently recreated identity. Etcd siblings also receive persistent changes through watches. Failed notifications/watches remain eventual propagation, not a distributed instantaneous revocation transaction.

Keep node clocks synchronized and monitor offsets. Ordering uses source wall clock timestamps, with monotonic advancement for local writes to the same path. It does not establish causal order between concurrent writers at different sites or resolve conflicting live updates with identical timestamps deterministically.

Users, child credentials and groups

A recreated user retains RevokedBefore. New service accounts and built-in STS issuances carry a signed siloParentRevocation claim identifying the parent boundary known at issuance. Editing or replaying an old child does not update this claim. Old children remain invalid even when their own update timestamp is newer than the parent deletion; children issued for the recreated parent remain valid. Claims are read from verified tokens on credential load/write.

A newer replicated service-account snapshot can replace an older service with the same access key, including a deliberate change of owner or secret. Local duplicate creates remain rejected. Snapshots preserve disabled status and the service’s own revocation boundary, so earlier mappings cannot attach to the new service. Equal-version retries reload the committed identity without rewriting it. Periodic live healing compares source versions even when the public status summary is unchanged, and includes disabled identities as healing sources. Service snapshots also preserve their absolute expiration. The receiving site does not reapply the local minimum lifetime for a newly issued credential. A newer already-expired snapshot still supersedes the old key, is denied by authentication, and is collected into a durable service tombstone by normal loading. Cache/claims loading failures are returned for retry, not acknowledged; after a committed replacement the stale cached secret is evicted immediately. Collisions with an existing built-in or cached STS identity report an error and require an explicit administrative resolution; replication cannot change its credential kind. Concurrent conflicting service updates with exactly the same timestamp can retain different winners at different sites; the status summary does not resolve that case.

Each group member has its own MemberGrants timestamp. Changing another member or the group’s enabled status does not reissue everyone else’s grants. Effective membership requires the grant to be newer than both the user’s and the group’s retained boundaries. Listings and policy evaluation use the same effective membership. Peer snapshots preserve grant times, including unknown legacy grant times; they cannot treat a recent snapshot time as a fresh grant to a revoked identity. A new explicit administrative group grant can restore access.

This does not implement a general conflict-resolution protocol for all group membership edits. In particular, the inherited live-group snapshot merge adds members and does not reconcile a missed ordinary member removal. Removing a member from a live group during a site outage is a separate known limitation; do not infer that this change resolves it. User/group deletion boundaries and same-name recreation are covered here.

Retention and expiration

Permanent identities, groups, policy documents and mappings have no automatic tombstone TTL. A disconnected peer or an old backup may return arbitrarily late. Successful replay acknowledgements are an optimization, not permission to garbage-collect this history.

Natural expiration of an immutable STS token physically removes its token-key record and any legacy token-key mapping, without generating a permanent tombstone. An early STS revocation is retained until that token’s expiration plus the existing clock-skew allowance. Replaying the same revoked token with a later event timestamp cannot recreate it. Etcd uses an expiration lease; object storage collects expired STS tombstones during its existing credential loading/purge. A record with unknown expiration is retained conservatively. Cleanup writes are best effort and use a short lock budget. If one fails, that load stops optional reclamation, reports the error and still loads healthy users; expired credentials stay denied and retain their existing durable version. The next load retries. Healthy cleanup has no fixed record quota. The reusable STS parent policy mapping is not assigned the token’s TTL by deletion cleanup. External-IDP disablement is an early revocation, not natural token expiration, and its cached STS and service accounts are included in cleanup. Expiring service accounts retain a durable revision because their access keys are reusable and an older version might have no expiration.

Direct per-token RevokeTokens delivery between sites is not a new guarantee of this change. The guarantee for built-in parent deletion follows from the durable parent boundary, including children not currently present in the deleting node’s cache.

Healing, failures and operational cost

Each process starts one healing loop. Losing its distributed leadership lease pauses work until leadership is reacquired; it does not permanently terminate healing. Configuration reloads do not create extra loops. The 30-second interval starts after leadership is acquired and after each completed pass. Initial lock retries and endpoint recovery can add further delay; it is not a convergence SLA.

The normal IAM loaders maintain an in-memory index of deletion records and retained boundaries, without secrets. Healing uses this index; it does not add a second full walk of config/iam/ every cycle. Existing full IAM loading still scans persistent records, including tombstones, at startup and on refresh.

Each peer receives batches of at most 128 records. The sender remembers which path/version each peer acknowledged. Unrelated new changes at either site do not reset that progress. A failed batch remains pending while later independent batches can progress; a lost response may cause safe idempotent replay. A pass has a bounded duration, and its successful acknowledgements survive that timeout. The protocol reports each node name and process instance. Switching between known node instances behind a load balancer preserves acknowledgements. A new node instance conservatively invalidates prior acknowledgements once; repeated switches among those known instances do not reset progress. Restore persistent state only with the affected processes stopped, so a restore cannot reuse an old process acknowledgement.

Steady-state healing still traverses/sorts the retained in-memory set and checks the peer’s protocol status. It suppresses repeated deletion PUTs once acknowledged. Memory use scales with retained paths and peers; startup storage reads scale with history. This release does not provide general history compaction. After upgrading, older live records whose receiving sites originally assigned different timestamps can require an initial reconciliation wave. Allow for its storage writes and sibling notifications when planning the maintenance window.

The cluster IAM metrics include revocation_records, revocation_heal_failures, revocation_heal_duration_millis, and revocation_heal_last_success_timestamp_seconds. Errors are also logged. A nominal 30-second scheduler interval is not a convergence deadline: outages, large backlogs, lock contention and failed requests can require more passes.

Cached credential lookup checks the in-memory parent revision index and performs no additional storage read. STS issuance and cold credential loading still consult the persistent parent revision. Revision I/O and distributed lock waits release the IAM cache lock while a separate local writer mutex preserves write order. These operations have bounded contexts, including etcd lock and lease cleanup.

Protocol and supported upgrade

The server-owned versioned route is /minio/admin/v3/site-replication/peer/iam-revisions. It carries source versions, member grant times and distinct user/group revocation items without changing the admin client SDK or S3 API. Older servers reject this route. The sender reports the failure and does not fall back to a route that would discard the metadata. The existing legacy IAM route remains readable for best-effort compatibility; this does not confer the new guarantees on an older peer.

All participating servers must be upgraded for the guarantee in this document. Mixed old/new nodes sharing an IAM backend and rolling downgrade are unsupported: older binaries do not interpret tombstones or signed parent boundaries correctly. Use a maintenance window for coordinated upgrade:

  1. Pause IAM changes and isolate any offline site or backup whose state is unknown.
  2. Back up each site’s complete IAM storage and required encryption material. A live IAM admin export omits deletion history and is not an adequate backup.
  3. Stop all nodes sharing each site’s IAM backend, replace their binaries, and restart them on the upgraded version. Complete this for every participating site before relying on the new revocation semantics.
  4. Check IAM loading, site-replication errors and revocation convergence. Verify representative old credentials are denied and deliberately reissued ones work.
  5. Resolve pre-upgrade revocations explicitly. Absence cannot reconstruct an already-lost deletion version: remove surviving old records on the sites that still have them, and rebuild stale offline peers from approved state before admitting them. Do not reconnect an unknown old snapshot just to discover its deleted credentials.

Credentials issued by an older server for a recreated parent lack the required signed boundary and must be reissued by an upgraded server. Parents without any retained revocation history preserve existing credential behavior.

A pristine built-in policy remains protected from local deletion. If an administrator explicitly overrides that policy and later deletes the override, the durable deletion now suppresses automatic recreation of the built-in policy on reload. This prevents reload from undoing the deletion. Restore the policy by an explicit policy-create operation if desired. Local deletion of a nonexistent policy remains idempotent and does not create a new tombstone; replicated unknown deletions retain their version.

For rollback, stop and isolate the affected sites and assess changes since the backup before restoring compatible state. Restoring an older backup can itself lose later revocations and requires reconciliation/rekeying before access is reopened. Do not delete tombstones online or convert only live IAM records to make an older binary start. Server, client, Console, package and deployment acceptance remain separate delivery gates of the maintained PGSTY stack.

Regression and performance checks

Focused coverage is in iam-revocation_test.go, iam-revision_test.go, iam-revision-lock_test.go, iam-revision-boundary_test.go, iam-replication-protocol_test.go, iam-credential-retention_test.go, iam-peer-reload_test.go, and iam-replay_test.go. Set SILO_TEST_IAM_REVOCATION_ETCD to a disposable etcd endpoint to include backend lifecycle/locking/boundary tests; they use isolated key namespaces.

BenchmarkIAMCachedCredential and BenchmarkIAMSetTempUser can be run against the pre-change source for a comparable local baseline. BenchmarkIAMRevisionConvergedHealing covers 1,000 and 10,000 retained records; it measures steady-state index/network work and asserts zero repeated PUTs. Its fake peer does not measure durable catch-up throughput. BenchmarkIAMColdLoadExpiredServices measures loading and cleanup with 100 or 1,000 expired reusable credentials; run it with -benchtime=1x. Use actual multi-site signed S3/STS tests and deployment-specific latency/scale measurements in addition to these component tests.

Errors and observability

An admin delete can return HTTP 500 after the authoritative revocation has committed but dependent cleanup failed. It is not evidence that the old identity remains usable. Retry the intended revocation while IAM changes and same-name recreation are paused, then verify old and reissued credentials at every site. When a persistent parent revision cannot be read or issuance cannot be committed, STS fails closed through an internal-error path, including STSInternalError; it must not mint a credential by guessing a missing boundary.

Scrape each process’s authenticated /minio/metrics/v3/cluster/iam endpoint:

Metric Interpretation
minio_cluster_iam_revocation_records Retained deletion records and parent boundaries in this process’s index
minio_cluster_iam_revocation_heal_failures Failed convergence passes since process start
minio_cluster_iam_revocation_heal_duration_millis Duration of the last pass
minio_cluster_iam_revocation_heal_last_success_timestamp_seconds Unix time of the last successful pass

The index is per process; summing it across siblings double-counts shared state. Healing requires site replication and a leadership lease. Shared-backend deployments without site replication do not run the site healing pass; zero healing metrics are not evidence of a failure there. A healthy counter on one leader does not prove every peer’s credential checks pass.

Why simpler alternatives fail

Physical deletion alone loses the ordering evidence an offline peer needs. Comparing only the child’s update timestamp is also insufficient: editing an old child after a parent deletion would appear newer and could restore access. The signed issuance boundary remains unchanged by such an edit. Replacing a received source revision with local receive time breaks source order and can turn a retry into a new mutation. The implementation preserves source timestamps and separately advances only deliberate local writes. Tombstone TTL or restoring only live export records discards the very history that prevents replay.

The versioned source and metrics definitions define this contract. This record replaces the former in-repository IAM design; it is not a production recovery certificate.

4.4 - Federated CopyObject: Preserve the Destination Contract

Source status, 2026-09-16: this records the main-branch CopyObject fixes in #157, #159, #163, #177 and #179. They are absent from Server 20260903. The subject is the legacy etcd bucket-federation path that forwards a copy to another deployment as PutObject, not the bucket/site replication scheduler.

Forward bytes once, encrypt at the destination

The proxy reads logical source bytes: it decrypts and decompresses as needed, then declares the logical length to the destination. It must not locally encrypt/compress and ask the remote to do it again. SSE-C reads still require the source key and secure transport. The destination’s explicit SSE headers or its own encryption defaults choose how the new object is stored; the proxy must not inject its local defaults when the caller selected none. SSE-KMS context is forwarded in the expected JSON-object form.

Trusted raw SSE-C replica CopyObject is explicitly rejected with 501 NotImplemented in this federation path, before creating the destination. That combination is not made safe merely by passing replication markers. Ordinary key-authorized SSE-C copies and the dedicated replica path are different operations.

Checksum and metadata rules

Remove every reserved internal metadata prefix, case-insensitively, from the ordinary forwarded write. Forward public metadata and tags through supported fields, not internal storage encoding. The checksum must describe the logical full object actually written:

  • A requested algorithm or a multipart-composite source requires a full-object checksum at the destination.
  • For a nonempty stream, the client uses a trailing checksum, including x-amz-trailer; the destination must consume and validate that trailer.
  • Empty content uses the ordinary checksum header because there is no streamed checksum trailer to carry its digest.
  • A stored full-object source checksum can be forwarded as an ordinary checksum header for validation.
  • A required remote checksum that is missing, malformed or has a multipart -N suffix is an error; it must not be reported as a valid full-object result.

The remote write may already have committed when a malformed success response is detected. A resulting error does not prove destination absence. Inspect the written version before blindly retrying a versioned copy.

Object Lock is not user metadata

Forward legal hold through the typed LegalHold option so it becomes x-amz-object-lock-legal-hold, not x-amz-meta-*. Retain the full precision of the retention timestamp; routing it through a whole-second conversion would weaken the requested value. Authentication and the destination’s Object Lock rules still apply.

Return the committed write

The response and ObjectCreated:Copy event describe the destination key, logical size, ETag and exact version ID. Modification time is obtained from the destination write, not invented at the proxy or obtained through a later unversioned HEAD that could observe another writer. The internal write-time response is bound to the authenticated federation request. Source-version response headers still identify the selected source when one was provided.

Evidence and deployment boundary

See the handler, write-time transport, and object-copy-federation*_test.go / object-federation-time_test.go. Tests cover ordinary, empty, multipart, compressed and encrypted sources, remote checksum failures, destination defaults, legal hold, response identity and events. These component fixtures do not certify a production etcd federation or every S3-compatible destination.

Verify both forwarding and destination builds from the component matrix. Existing objects are not rewritten by upgrading. Related records cover SSE-C replica integrity and Object Lock ordering.

4.5 - Multi-Pool Object Consistency

Source status, 2026-09-16: #178 repairs multi-pool mutation ordering; #207 extends current-object selection to conditional PUT. These changes are merged into main and absent from Server 20260903. They close the source defects tracked by #133 and #144; this does not establish acceptance of a new release artifact.

One key can have several physical copies

A pool expansion, decommission or interrupted cleanup can leave copies of one key in different pools. A per-set lock does not serialize an operation with a writer choosing another pool. Nor does reading the first copy prove that its ETag, tags or retention represent the logical object.

The relevant failures include a stale ETag satisfying a conditional write or delete, a newer retention/hold disappearing behind older object metadata, and an older copy becoming visible after deleting the selected copy. Reading each pool independently and combining only its success status does not solve these races.

Shared selection and lock boundary

The pool coordinator holds one namespace object lock across selection and mutation. objectPoolInfos reads the addressed version in every pool, including draining pools. Unknown/unreadable state is an error, not evidence that the object is absent. Copies are ordered by modification time with pool index as the equal-time tie-breaker.

An unqualified metadata request first resolves the logical current version, then gathers copies of that version. It must not merge metadata from unrelated object versions. Object Lock retention, legal hold and tags have independent source timestamps; mergedPoolObjectInfo combines those fields by their own order rather than treating the object’s modification time as every field’s revision. An ordered removal is state too.

Metadata writers, relevant object writers and healing use the same coordinating lock. The lock is not a transaction that can undo completed disk writes on every pool after a later failure.

Conditions and deletion

For conditional DELETE, evaluate once before cleanup and clear the callback before invoking lower pools. An explicit versionId compares that version. For conditional PUT, select the latest logical representation across eligible pools while holding the outer lock, including the delete-marker state, before installing the replacement. #207 addresses the case where a stale destination-pool copy made If-Match/If-None-Match disagree with the current object.

Cleanup and metadata propagation can still partially modify physical copies before a storage failure is reported. A quorum/cleanup failure must not be interpreted as an unchanged object or as proof every old copy disappeared. Verify state and retry after recovery. Batch XML ETag conditions remain unsupported; see conditional DELETE.

Object Lock and replication

Replica writes reconcile destination retention and legal-hold state against authoritative same-version copies across pools. A stale incoming replica must not shorten a newer retention or turn off a newer hold. Tags similarly travel with their timestamp, including an empty deletion value. See Object Lock ordering and replicated tags.

Evidence and operator impact

The coordinator implementation, pool entry points, and the two merged PRs identify the source contract. Coverage includes multiple pools, stale copies, delete markers, explicit versions, concurrent writes and failure paths. It is not a claim of atomic rollback or arbitrary fault tolerance.

Upgrade all participants before depending on the shared-lock guarantees. Inventory historical copies, tags and Object Lock state when a deployment may already have encountered the defect; the source repair does not prove historical cleanup. Use the replica audit runbook and component matrix. The multi-pool repair is separate from the default limitations and scan cost of multipart upload listing.

4.6 - Object Lock Replication Ordering

Release boundary, 2026-09-16: f4c1286c9 shipped in Server 20260903. The later timestamp-only removal, SSE-C retransmission and cross-pool repairs in #129, #134 and #178 are on main and not in that release. The advisory ledger records the original ordering defect.

Preserve the old state before rebuilding metadata

A replica COPY used to rebuild metadata from the incoming request before comparing retention and legal-hold timestamps. The old timestamps were therefore lost, so a stale request could appear authoritative. The legal-hold timestamp also went into the retention timestamp key. The first fix captures the stored state before rebuilding the destination map and keeps each field’s revision under its own key.

Retention and legal hold are independent registers. A later retention does not make an older legal hold authoritative, and the object’s modification time is not a substitute for either field’s revision. Apply an incoming field only when its timestamp is newer than that field’s stored timestamp; retain the stored value for a stale or equal update.

Removal is an ordered value

Removing retention or a hold can leave no live value but still carries a source timestamp. Dropping that timestamp would let a delayed old value return. The later repair recognizes timestamp-only removals during receive, comparison and resend. An absent value with no ordering evidence is different from a recorded removal.

The receiver must preserve these distinctions through ordinary metadata copies and full SSE-C replica retransmission. A retransmitted object cannot erase a newer destination hold or resurrect an older retention just because its body is being rewritten.

Cross-pool authority

If one version has copies in multiple pools, one local set cannot decide the latest field state. The multi-pool coordinator gathers same-version copies under the shared object lock and merges retention and hold independently by their timestamps. Unknown pool state is not an empty value. This closes the source-level boundary originally left open as #133.

Authorization and limitations

Ordering does not grant permission to change Object Lock. The replication trust checks and the relevant S3/admin permissions still apply. It is not a new user API for bypassing governance or compliance retention. It also does not create causal ordering between unsynchronized source clocks or a distributed rollback after partial storage failure.

A historical stale update may already have changed stored state. Upgrading only prevents the repaired paths from accepting the same error again; it does not reconstruct missing retention history. Verify exact-version retention and legal hold at each site against authoritative records before declaring recovery.

Evidence

See the Object Lock merge implementation, the replica write paths in erasure-object.go, and the linked PRs. The tested cases include stale/newer field updates, empty-value removals, distinct field timestamps, SSE-C retransmission and multiple pools. These source tests do not prove every historical replica has converged.

Use the replica audit runbook, SSE-C integrity record and release matrix together. Upgrade all participating nodes before relying on the combined ordering contract.

4.7 - SSE-C Replica Integrity

Source status, 2026-09-16: the repairs in #122, #123, #124, #126 and #134 are on main, not published Server 20260903. Earlier zero-byte/read-attribute authentication and destination-key checksum repairs did ship in 20260903. Do not treat all SSE-C fixes as one release.

Ciphertext and trust

An ordinary SSE-C request supplies the customer key and operates on plaintext. An authorized replica transfer can carry raw ciphertext with the sealed object-key metadata needed to preserve the original object. The destination must store those bytes verbatim; encrypting them again produces an unreadable double-encrypted object. The internal marker alone is not authorization: the exact replication marker and the required replication permission remain mandatory.

This path does not reveal the customer key or make a normal keyless read permissible. Ordinary GET/HEAD and GetObjectAttributes keep their key/permission checks. Federated raw SSE-C replica COPY is explicitly unsupported.

Multipart sizes and checksums

Encrypted part size and logical plaintext part size are different. Replica multipart records preserve the actual logical size of each part so partNumber reads, ranges and GetObjectAttributes agree with the source after overwrite. Checksums must retain the correct encryption context and logical meaning; a checksum response cannot be decrypted with the source key after a copy committed under a different destination key.

For ordinary SSE-C key rotation, any explicitly requested checksum algorithm forces a complete rewrite on current main, even if it names the existing algorithm. A multipart source becomes a single-part destination; the ETag can change and replication retransmits object bytes. An eligible metadata-only rotation without that request preserves the prior checksum state, including absence. See the operator procedure.

Retransmission and Object Lock

A keyless target HEAD can report an existing SSE-C object as inaccessible rather than absent. The sender must distinguish this from NoSuchKey; it cannot assume an ordinary metadata-only COPY will repair the replica. Existing SSE-C replicas use object retransmission. A destination with undecodable old replica state can be replaced through the repaired retransmit path, while preserving the newer destination Object Lock state and ordering removal timestamps correctly.

A failed old replica is not automatically certified repaired after a Server upgrade. Re-read the exact source and replica versions with approved key access, compare the logical bytes and part boundaries, and check retention/legal hold separately. Never infer integrity from a successful metadata-only HEAD alone.

Compression and historical objects

Current main excludes every new SSE-C write from compression, including normal PUT, multipart initiation, COPY and Snowball. SSE-S3 and SSE-KMS continue to follow their encrypted-compression setting. This avoids a replica format that transports ciphertext without the required compression metadata.

Historical compressed SSE-C objects are not automatically rewritten. Preserve their keys and exact version identities, inventory affected data, and rehearse a supported rewrite/recovery path. The change is preventive, not a background migration or a promise that arbitrary old ciphertext can be recovered.

Verification boundary

The compression decision, replication-trust-ssec-replica_test.go, erasure-multipart-ssec-replica_test.go, replication-ssec-retransmit_test.go and compression-ssec_test.go preserve the relevant source and regression contracts. Tests include incorrect-key/unauthorized controls, multipart byte comparisons and retransmission cases. They do not replace a deployment’s historical-state inventory.

See Object Lock ordering, multi-pool consistency and replica recovery. Upgrade every participating Server before relying on the combined behavior.

4.8 - Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation

This page records the analysis, design choices, review, and implementation of #153, #152, and #137. They belong to the same replication reliability series, but affect operation classification, recovery visibility, and task lifecycle respectively. One general retry patch cannot repair all three.

As of 2026-09-09: PR #162 is merged as d1105bbb, and all three issues are closed. All eight checks on the tested PR head, followed by main Go CI and VulnCheck, passed.
Second round, 2026-09-16: PR #196 (fix 0c61128d2, verification record aea3882c9) repaired the replication worker’s own delete exits and the persisted MRF marker-recovery path — a different surface from the first round. It is on main and not in Server 20260903; see the second-round section.
Review: the plan was discussed with the installed Claude Code Fable 5.1 Max, followed by a review of the implementation. The final verdict was GO.
Delivery boundary: this work completed code, tests, and main integration. It did not create a Server tag or formal release, and does not establish that existing packages, images, or production deployments contain the fixes.

Overall decision and the surrounding series

The selection rule was to repair a reproduced invariant at the smallest boundary that owns it, retain existing recovery mechanisms, and use deterministic tests to prove that work can finish. Additional complexity needs a concrete counterexample.

Issue Confirmed defect in current SILO Selected repair
#153: delete-marker purge Single-object DELETE classified a permanent deletion as marker replication; the remote marker disappeared while source purge remained PENDING Classify by purge status, matching bulk delete, scanner/heal, and resync
#152: invisible MRF drops The queue was already bounded, but internal drop counters did not reach administration or monitoring; object and delete worker arguments were reversed Expose existing counters, warn at actual drop sites with deduplication, and align worker routing
#137: unreliable resync cancellation One shared token could not stop multiple runs; blocking phases missed cancellation; stale runs could overwrite terminal state Give each run an owned context, cancel by resync ID, and constrain registration, finalization, and state updates

The surrounding series had already separated three concepts that are easy to confuse:

  • #136 / PR #138 repaired counter completeness: receive and apply the final result before persisting terminal state, without waiting for the one-minute periodic flush.
  • #139 repaired outcome accuracy: an existing destination object does not prove that this update succeeded. Success and failure must come from the target’s actual replication result.
  • #137 repairs cancellation and resource lifecycle: the task must stop, its walker, workers, and result consumer must exit, and an old run must not pollute a new run’s state. This change preserves the first two contracts rather than introducing another accounting mechanism.

Authorization to use internal replication semantics belongs to the earlier CORS and replication trust record. Authoritative Object Lock state across pools remains tracked separately in #133, open at the original record date and subsequently repaired on main by #178 (not in Server 20260903); see multi-pool consistency. Closing these three issues does not mean every replication concern is resolved.

The release-gating target is the maintained pgsty/silo stack with Console, mcli, and silo-pkg. Compatibility with upstream MinIO/MC remains best effort. Neither a mechanism proposed for upstream nor an external experiment can automatically be treated as an observation on current SILO.

#153: distinguish marker replication from permanent version deletion

The report and the reproduction differ

The original issue described a sustained HTTP 405 storm involving ILM and replication, and proposed treating every delete-marker 405 probe as completion. Current SILO source and measurements do not justify adopting that explanation and patch directly.

The baseline was 450dcb848. The experiment used two locally built SILO servers, separate disposable data directories, and maintained mcli / minio-go clients. A critical observation was that mcli uses bulk DELETE even for a single key; that route was already correct. A direct single-object S3 DELETE was therefore necessary to exercise the faulty entry point.

Observation Single-object DELETE before the fix After the fix
Matching delete-marker version at the target Removed Removed
Purge state for that version in source xl.meta Still PENDING, although ordinary replication status was complete Cleanup completed on the first replication attempt
Original data version Retained Retained
Scanner needed to finish this cleanup Required later recovery No scanner assistance needed in this experiment

Checking only that the remote version disappeared would falsely declare success. Source metadata must be inspected too. The reproduction establishes incorrect initial purge classification and completion state, plus an unnecessary probe. It did not reproduce the external report’s sustained 405 storm or request-volume figures. Existing scanner/heal and resync producers already select the proper purge path; they cannot be described as necessarily repeating that erroneous probe every cycle.

Why one classification condition is sufficient

When deleting an existing marker, the object layer can return both DeleteMarker=true and a nonempty VersionPurgeStatus. The former describes the version being operated on; the latter describes the operation now required. They are compatible facts.

The old single-delete handler checked only DeleteMarker and populated DeleteMarkerVersionID. Completion then updated ordinary ReplicationStatus instead of completing the purge. The final condition is:

if objInfo.DeleteMarker && objInfo.VersionPurgeStatus.Empty() {
    dmVersionID = objInfo.VersionID
} else {
    versionID = objInfo.VersionID
}

This matches the classification already used by other producers. The fix belongs in the DeleteObject handler. It requires no storage-format change, relaxation of replica deletion protection, or new recovery task. Legacy PENDING entries remain recoverable through the existing scanner/heal purge path.

Why a 405 is not unconditional success

A 405 from a versioned HEAD can establish that the marker exists at the target. For replicating a delete marker, that can mean idempotent completion. For permanently deleting that version, it means there is still work to do.

Writing VersionPurgeComplete merely because HEAD returned 405 could let the source remove its metadata while leaving the unwanted target version behind. The implementation retains existing 405 semantics. Real remote failures such as 403, 405, and 503 must not be disguised as successful permanent deletion.

Source history traces marker-based classification to an upstream 2020-11-19 commit. The 2023-07-10 optimization changed scheduling from dsc.ReplicateAny() to the returned object’s replication/PENDING purge state while retaining classification based only on DeleteMarker. At this location, the 2025-04-02 commit merely moved Pending to replication.VersionPurgePending; it did not first introduce that scheduling condition. This identifies lineage, not a bisect across every historical release, and does not establish when the entire externally reported storm was introduced.

#152: expose drops from an already bounded queue

MRF, or Most Recent Failures, records recent failed replication work for background processing. It already had limits: mrfSaveCh has capacity 100000, and mrfRetryLimit is 3. The code drops a queue entry when RetryCount > mrfRetryLimit or the save channel is full.

The source object and its pending replication state remain. Dropping a queue entry does not mean losing source data. However, subsequent repair depends on the scanner; a prompt retry can turn into a wait for a scan, so this mechanism does not establish a fixed recovery deadline.

The actual defect was that TotalDroppedCount / TotalDroppedBytes increased internally but were omitted from both public statistics snapshots and from Prometheus v2/v3. A deterministic capacity-one fixture demonstrates the failure: one entry is admitted, a 20-byte overflow entry and a 30-byte retry-exhausted entry are dropped, and internal totals become 2 / 50 while administration reports 0 / 0.

The selected minimum change

  1. Atomically read the existing counters into both administration snapshots.
  2. Register and load cumulative counters in metrics v2 and v3, and document their meaning.
  3. Warn at the two actual drop sites: retry exhaustion and a full MRF channel. Fixed messages and deduplication keys prevent changing counter values from defeating deduplication. Handing ordinary worker overflow to MRF is not itself a drop, so it does not gain a warning here.
  4. Route ordinary objects, healing, and deletion consistently by (bucket, objectName) to restore worker affinity for the same object.
Interface New metric
Prometheus v2 minio_node_replication_mrf_dropped_operations_total
Prometheus v2 minio_node_replication_mrf_dropped_bytes_total
Prometheus v3 minio_replication_mrf_dropped_operations_total
Prometheus v3 minio_replication_mrf_dropped_bytes_total

These counters accumulate since Server startup and reset on restart. Operations count entries, not unique objects; one object can contribute repeatedly. Bytes cover known sizes, with deletion entries contributing zero bytes. They measure neither data loss nor the complete backlog. Operators should examine increases alongside replication backlog and target health.

This work does not enlarge the queue, increase retry limits, add a persistent retry scheduler, or introduce another backoff framework. Existing limits continue to bound memory use, and the scanner remains the eventual repair mechanism. The repair makes an invisible condition observable; it does not promise a recovery deadline under arbitrary failures.

#137: make cancellation part of a run’s lifecycle

One token cannot cancel a group

Previously, cancellation placed one unkeyed token into a shared channel. One site resync can cover several buckets, with up to 10 concurrent bucket runs, while dispatchers and workers compete to consume the token. Only one bucket might stop; an unrelated task might consume it; or a leftover token might affect a later task.

Two blocking phases had independent defects: a bare receive from Walk output did not observe cancellation, and sending to a full worker channel did not observe it either. Once a worker exited, the dispatcher could block forever on a channel with no receiver. Walk also inherited the parent context, so an early function return could not stop its own walker.

State handling had related faults: site updateState modified a local value without writing it back to the map; bucket Canceled handling was incomplete; and an old finalizer could overwrite canceled state with Completed.

Registration, cancellation, and status share a write boundary

Each resyncBucket creates an owned context.WithCancelCause and registers before waiting for a concurrency slot. Registration and cancelResyncID share the resyncer’s state lock:

  • Registration validates that the target still exists and the resync ID still matches, and reads its current cancellation state.
  • Cancellation marks matching Pending / Started states Canceled before canceling all matching registered contexts.
  • A registered run still waiting for a slot receives cancellation; a run registered after cancellation sees the canceled state.
  • Walk, dispatch, workers, and result sends observe that run’s context. Unrelated IDs are unaffected, and there is no buffered token for a later task to consume.

A dedicated operation mutex serializes site start/cancel configuration setup. Canceling running work still executes if part of the target-configuration loop fails. Bucket finalizers and counter updates check the current target and resync ID, ignoring late results from removed, replaced, or canceled runs. Site state is actually written back, and a late Completed result cannot overwrite Canceled.

Success and failure require different finalization order

The result-consumer contract from #136 must remain intact: persist terminal state only after the consumer finishes.

Success: close worker inputs → join workers → drain results
         → persist terminal state → return slot → cancel owned context and unregister

Failure/cancellation: cancel context first to unblock work → join workers and consumer
                      → persist the appropriate terminal state → return slot → unregister

Canceling before joining workers on a normal completion path could discard pending work and turn a successful run into Failed. The final code calls finish exactly once from one defer, using defer order to complete cleanup. It therefore needs no additional sync.Once.

WithCancelCause distinguishes an explicit user cancellation from parent-context interruption. User cancellation becomes Canceled; parent interruption observed during finalization must not leave an interrupted run marked Completed. Shutdown while a run is still waiting for a slot preserves the existing Pending state for restart recovery.

Protect terminal state and resumed work

Cancellation can still arrive between the context check and the terminal save. Under its lock, markStatus must persist an existing Canceled state even if the finalizer previously computed Completed. An old resync ID must also be unable to overwrite a new run.

This is not a transaction across metadata files. If a bucket completed and persisted before cancellation, a subsequent site cancellation can leave different records reading “bucket Completed, site Canceled.” Completion happened first. The guarantee is that late completion cannot turn an already canceled run back into Completed, not that cancellation erases work completed before it.

The recovery loader, loadResync, previously launched goroutines and then immediately executed defer cancel(). Inspection of SILO’s actual shared-lock implementation confirmed that this cancel ends the merged leader context; it is not a no-op. A WaitGroup now retains that context until resumed runs exit. Losing leadership still cancels the existing context; replacing it with a global context would bypass the leadership constraint. Loading disk state must also preserve newer in-memory start/cancel state.

Fable review and the complexity decisions

The review used the installed Claude Code with model argument claude-fable-5-1[1m] and --effort max. Baseline reproductions and the minimal proposal produced agreement on all three repairs. The final patch and validation were then reviewed again, yielding GO. The conclusion rests on code and evidence, not model agreement alone.

Proposal or review point Final decision
Treat a purge’s 405 probe as completion Rejected: marker existence does not prove permanent deletion
Add bounded retries, backoff, and a persistent MRF scheduler Not introduced: the existing queue and scanner already provide recovery; the demonstrated gap is visibility
Send more shared cancellation tokens Rejected: this still cannot guarantee identity routing, broadcast, or isolation from later tasks
Add a separate cancellation tombstone registry Unnecessary: existing target status and resync ID under one lock close the registration race
Give each run an owned context and cancelable blocking operations Retained: full-queue deadlock and walker leaks have deterministic reproductions
Protect finish with sync.Once Initial review required protection against double close; the final implementation has one deferred call site, and re-review accepted omitting Once
Wait for resumed runs before releasing leader context Initial review required checking necessity; SILO’s cancel is effective, so the WaitGroup stays, with leadership-loss coverage

The final review accepted two further implementation boundaries. The infrequent resync start operation holds the status lock across configuration reads and writes, consistent with existing terminal saves. The active registry uses resyncOpts, including resyncBefore, as its key; current callers reuse the same in-memory values, and no identity mismatch was reproduced. Future changes that reconstruct time values or restore task identity should revisit equivalence, rather than adding another registry without evidence now.

Validation and reproducible evidence

Regressions were added against unchanged production code first, and failed before the fixes. New tests exercise the production handler, real erasure storage, actual metric registration, and resyncBucket, rather than only testing helper logic that repeats the implementation.

Validation area Result and evidence
Single-delete purge ErasureSD and Erasure source/target cleanup; legacy PENDING recovery; marker idempotency; real 403/405/503 failure semantics
MRF Actual capacity-one overflow and retry exhaustion; administration JSON; registered v2/v3 counter types and values; object/delete worker affinity
Resync testing/synctest coverage for blocked Walk, full worker queues, user and queued cancellation, unrelated IDs, later tasks, terminal races, stale IDs, slot release, and leader recovery
Complete local suite go test ./... -count=1 -timeout=30m passed, with 50 tested packages
Concurrency and repetition Focused replication race tests passed; cancellation regressions passed 100 repetitions
Tooling and contracts Local build, vet, lint, generated-file checks, rebrand compatibility guard, and git diff --check passed; dependencies and compatibility baseline unchanged
Two native servers Built from local source; direct single-object DELETE removed the target marker and source xl.meta entry while retaining the original data version; no downloaded Server Docker image
Remote integration All eight PR checks passed, followed by all six main Go CI jobs and VulnCheck

Tests pinned to the merged revision: delete markers, MRF visibility, and cancellation lifecycle. With that revision and Go 1.27.1, the relevant checks can be repeated with:

go test ./cmd -run '^(TestReplication|TestReplicateDeleteMarker|TestResync|TestSiteResync)' -count=1
go test -race ./cmd -run '^(TestReplication|TestReplicateDeleteMarker|TestResync|TestSiteResync)' -count=1
go test ./cmd -run '^TestResyncCancel' -count=100
go test ./... -count=1 -timeout=30m

After complete local validation, only new-test formatting and fixtures changed: an existing ARN was reused, and the collector supplied the metric prefix, preventing the compatibility scanner from treating test strings as new protocol identifiers. The guard was not weakened. Relevant tests, lint, and compatibility checks were rerun after those adjustments, and remote CI checked the final commit.

Evidence point Exact identity
Baseline 450dcb8484bc1337deba0cf608cc893a6691d794
Final PR head 66fe61ff65c83d68b74baa637a11623015c7aa21
Merged main d1105bbb3d4a0afa33b3a4ac11b821235038ed0e, with the same source tree as the final PR head
PR Go CI 34320440012
Main Go CI 34321319278
Main VulnCheck 34321319274

Maintenance and release decisions

Future changes must continue to establish operation identity, visible failure, truthful per-object outcomes, complete terminal counters, and cancellation that releases its own resources. Neither an API returning Completed nor an object existing at the destination can replace those checks.

The code verdict is GO, and the delivery facts are main integration and passing CI. A formal release still requires selecting a tag, verifying packages and images, and establishing that deployments contain the repair. The scanner-dependent MRF recovery delay, the open cross-pool issue, and the externally reported 405 storm not reproduced on current SILO remain part of this decision record.

Second round (2026-09-16): worker-side purge classification and persisted MRF recovery

The first round classified deletes at the handler entry (#153), exposed MRF queue drops (#152), and completed resync cancellation (#137). The second round — PR #196, fix 0c61128d2, integration verification aea3882c9 — repairs a different surface: the replication worker’s own exits, the outer aggregation, and the persisted MRF recovery path for delete markers. The two rounds are complementary; neither subsumes the other.

Status: on verified main 40220bd836cb, not in Server 20260903. All evidence is synthetic (real single-drive and 16-drive storage, signed DELETEs, controlled HTTP targets, real persisted-MRF disk files replayed through a fresh worker pool). The externally reported 405 storm is not reproduced and not attributed to these paths.

What was still broken

  • Legacy-shape tasks were skipped entirely. A task with an empty VersionID, a non-empty DeleteMarkerVersionID, a COMPLETED creation target, and a PENDING purge target never issued its DELETE — the per-target creation early-return suppressed it.
  • Fixing only the target function made aggregation lie. The outer status selection keys on VersionID and creation state, so a failed legacy-shape purge aggregated as COMPLETED, emitted ObjectReplicationComplete, and skipped queueing to persisted MRF — worse than the baseline.
  • Persisted MRF dropped every marker 405. Replaying a disk MRF entry for a delete-marker version fetched real marker metadata plus MethodNotAllowed, and the error path discarded it — the marker MRF recovery route was a dead end.
  • Failed purges overwrote successful resync markers, completed purges were re-sent, and a not-yet-ready HEAD unconditionally overwrote creation state.
  • A multi-target empty-state regex misparse could corrupt creation state once. A disk marker carrying only creation metadata, replayed against a task with purge state, let two empty target states parse as a bogus Pending; the deleted-flag guard then rewrote the whole creation block as empty entries with fresh timestamps — a one-shot corruption that requires disk/task divergence to reach.

Root causes: the worker had no task-level purge classification (the per-target predicate in the community’s #184 was itself wrong — a composite purge status can never reach COMPLETE through it; its investigation and proposed fix nevertheless shaped this follow-up, with thanks to Julien Laurenceau); MRF recovery had no valid-405 identity gate; and the delete task carried no retry count, so the existing mrfRetryLimit drop was unreachable on the delete path.

The repair

  • One classification, every exit. isVersionPurge() (non-empty VersionID, or a non-empty DeleteMarkerVersionID with a composite purge status) drives both the inner target function and the outer aggregation. Purge exits write only VersionPurgeStatus and leave ReplicationStatus empty — the storage layer’s “do not update” signal.
  • A three-field clear guard empties the composite status on the purge path, making the multi-target misparse unreachable at disk writes.
  • Purges send the canonical permanent-delete request — explicit versionId, ReplicationDeleteMarker=false, no HEAD/readiness probe (authorization is the DELETE’s own). This also prevents a lost-response retry from re-creating a marker at an unversioned fallback.
  • A valid-405 gate for MRF recovery. A MethodNotAllowed schedules recovery only when the returned object is a delete marker, the bucket/object/version identity matches, and the modification time is non-zero.
  • A bounded retry budget. Delete tasks carry a retry counter, incremented at all three persisted-MRF entry points (aggregation failure, lock failure, queue-full fallback), respecting the existing limit; after exhaustion, the scanner can still re-raise healing.
  • Audit and event status map COMPLETE to COMPLETED at the statistics/event boundary only, reusing the existing legacy constant.

What 405 means, precisely

For creation (replicating a delete marker), a HEAD 405 on the target marker version means already created — idempotent completion. For purge (permanently deleting a version), a 405 from MRF identity probing with full marker identity means work remains; the purge’s own success is decided by the DELETE alone, and a DELETE 403/405/503 is always a real failure. Empty or null version markers return ObjectNotFound, not 405, and sit outside the gate.

Boundaries that remain

The legacy in-memory task shape does not serialize across restart and has no current producer — its handling is robustness, not an active repair. Targets without a configured client still only log. Target-level resync replacing a purge subset is a pre-existing defect this round neither caused nor fixed. Persistence was driven directly in tests; timer-based flush and process-crash durability are not claimed. Multi-process site-replication meshes and cross-region acceptance are out of scope. For the tag-ordering and replica-metadata repairs in the same reliability series, see Replicated Tag Ordering and Replica Metadata Normalization.

4.9 - Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection

This page records the analysis and repair of two defects in how object tags keep their ordering across replication, merged into Server main as PR #193 (fix 03027727d) and PR #196 (fix 680eac66e).

As of 2026-09-16: both fixes are on verified main 40220bd836cb. They are not in the published Server 20260903; use the linked PRs to identify a build containing them.
Delivery boundary: source acceptance (regression tests plus the R4–R8 integration run whose PR #196 checks passed) only. No tag, package, image, or production rollout is established by this record.
Evidence class: synthetic signed HTTP tests use real single-drive, 16-drive and multi-pool erasure backends; sender, wire-shape and precondition checks also include function-level tests. No customer incident is attributed to these paths.

The shared model: a tag value and its revision are one state

Tags replicate with an internal revision timestamp (wire header X-Minio-Source-Tagging-Timestamp, stored as x-minio-internal-tagging-timestamp). The ordering rule on the receiving side is simple: a replicated tagging state wins only when its revision is newer than the stored one. Both defects in this family break that rule by making the revision — not the value — the part that gets lost:

  • R4 dropped the timestamp on the way into an SSE-KMS destination, so newer tag updates lost to stale stored state inside the storage-layer reconciliation.
  • R5 meant an empty value (a deletion) carried no revision at all, so the protocol could not express “deleted at time T” — and a delayed event could resurrect what a client had already deleted.

R4: SSE-KMS destination copies dropped tag revision timestamps

Failure form. A trusted replication COPY to an SSE-KMS encrypted destination returned 200, the object was correctly encrypted, plaintext GETs worked — and the destination’s tags and their timestamps stayed at the old values. Because the HTTP request succeeded, nothing surfaced the loss. The storage-layer reconciliation (reconcileStoredObjectTags) compares revisions under the object write lock; without the incoming timestamp, the old stored state won.

Trigger surface. Not only explicit SSE-KMS headers. Bucket-default KMS and global automatic encryption hit the same code path, so the defect could fire with no KMS header in the request at all.

Root cause. The option builder for PUT-like requests parsed the trusted source-tagging timestamp into ObjectOptions, then the SSE-KMS branch constructed and returned a different ObjectOptions carrying mtime, ETag, replication trust, and two Object Lock timestamps — but not ReplicationSourceTaggingTimestamp. The omission dated back to upstream c4373ef290 (2021); a 2026 Object Lock repair added two more timestamps to that literal and still missed this one. Before R5, the consuming path was COPY ordering. R5 adds timestamp persistence for replica PUT and multipart initiation, as well as the duplicate-precondition consumer. Those SSE-KMS paths also depend on R4 preserving the option, so backports must consider the pair together.

Fix. One field added to the existing SSE-KMS ObjectOptions literal (03027727d), nothing else. Regression tests cover every destination encryption (none, SSE-S3, SSE-KMS with and without key context, SSE-C) crossed with trusted/untrusted source, missing/valid/malformed timestamps, and 50 signed ordered COPY+GET cycles across two single-pool backends (single-drive and 16-drive erasure) with 1–3 ns event spacing. The KMS cases use a test stub.

No backfill. A lost source-tagging timestamp cannot be reconstructed at the destination. After upgrading, new tag events replicate in order; replay of old events still follows the timestamp comparison: an incoming event must be strictly newer; the stored value wins ties and rejects older events.

Deferred observation. The same SSE-KMS literal also omits the proxy and speedtest option fields; the speedtest flag is read on the storage path, so global auto-encryption would drop it on a speedtest PUT. Recorded here as a separate follow-up, without asserting a public issue exists; deliberately not bundled into this repair.

R5: empty tag values had no revision, so deletions could resurrect

Failure form. Nine baseline regressions across real-storage and function-level tests:

  • A successful DeleteObjectTagging never minted a new revision, so a later-arriving trusted metadata COPY with an older view of the tags re-instated them.
  • A newer COPY carrying an empty tagging state was ignored — empty meant “nothing to say” instead of “deleted”.
  • The first replica PUT parsed the source tag timestamp and never persisted it.
  • Equal visible tag values collapsed to “no replication needed”. HEAD does not expose a tag revision, so a newer deletion or re-addition was invisible and ordering was not restored.
  • A queued replication event’s completion callback wrote the old tags from its snapshot back over an already-committed deletion — and without a timestamp, the resurrected set inherited the deletion’s newer revision, which is worse than the originally-reported symptom.

Root cause. A tag value and its timestamp (including the empty value’s timestamp) constitute one state. The old protocol could only represent non-empty states: DELETE-tagging never minted a revision, the sender only attached timestamps in the non-empty branch, and the receiver only made decisions in the non-empty branch.

Fix (680eac66e):

  1. Every tagging mutation mints one revision. PUT/DELETE tagging handlers unconditionally stamp a single UTC RFC3339Nano revision, whether or not replication selects the object. The storage layer enforces monotonicity under the write lock: a local revision that is not strictly newer is advanced to stored+1 ns, and multi-pool backends compute one value that strictly exceeds every pool’s copy.
  2. The sender transmits tombstones. Empty values carry their recorded revision; empty-without-revision is not fabricated; a malformed recorded timestamp fails closed rather than being silently repaired.
  3. The receiver accepts tombstones. Replication COPY captures the stored tag pair before rebuilding, so an empty value with a timestamp enters the existing reconciliation as a state that can win. Duplicate suppression is relaxed only for a strictly newer source revision. Multipart completion orders tags from the persisted upload metadata. The delete-acknowledgement path no longer writes snapshot tags back. Ordinary SSE-C rotation drops the old tag timestamp from copied encryption metadata so it cannot overwrite the new local revision.

Operator-visible changes:

  • Equal timestamps now resolve to stored-wins on unqualified and explicit null COPY requests. The old handler let the incoming state win ties on the unqualified path; the change is a compatibility-visible tightening in the correct direction.
  • An ordinary COPY with tagging REPLACE and no tags now genuinely clears destination tags. The old default-metadata path carried source tags over — an S3 consistency improvement, but a behavior change.
  • Every ordinary COPY (including key rotations that change nothing) records one new local revision.
  • Both ends of a replication pair must upgrade together. An old peer still drops empty-value revisions, so a new sender’s tombstones are invisible to it.
  • Objects with recorded revisions incur one extra metadata COPY per object during explicit resync/heal. When the destination has bucket-default or automatic KMS encryption, that “metadata” COPY rewrites the object data — budget accordingly for bulk resync.

A nonempty legacy tag set without a revision uses object ModTime as its sender fallback; an empty set without a revision does not acquire a fabricated tombstone. A strictly newer trusted tag revision bypasses only the internal ETag/version duplicate guard (otherwise 412 PreconditionFailed); client If-Match and If-None-Match remain enforced.

The tag repair adds no wire field or storage format and has no capability negotiation. Both endpoints are needed for ordered deletions; an old hop retains the old behavior. This statement does not authorize rolling upgrade or downgrade of the entire September candidate, which also contains the separate IAM migration. Reconcile already-damaged tags from an authoritative source by a new explicit tag change; the lost historical order cannot be reconstructed.

Limits, stated as limits:

  • No migration: history with missing or wrong deletion timestamps cannot be reconstructed, and no historical tombstones are fabricated.
  • Replication rules with tag filters evaluate target eligibility against the post-deletion (empty) tagging state, so a rule filtered on the deleted tag never sees the deletion. This is a pre-existing scoping decision, unchanged here.
  • Arbitrary clock skew is not a total order. The repair establishes per-hop ordering, not multi-site causality.
  • A malformed recorded timestamp makes the sender’s construction fail permanently and the event retry through MRF until an explicit, correct tag change replaces it — deliberately fail-closed.

Rejected alternatives

  • Synthesize a tombstone from ModTime for empty-without-revision objects. Every object that never carried tags would gain a revision; combined with forced metadata replication, every object would take the metadata-COPY path on every hop.
  • Transmit only when the value is empty. Withdrawn by its own proposer during review: the equal-value re-add sequence (set X at T1, delete at T2, re-add X at T3) loses the re-add’s revision under that condition. A regression test (TestTaggingRepeatedValueNeedsRevisionDelivery) pins the counterexample.
  • A new HEAD revision protocol. The pinned minio-go metadata extractor discards internal response headers, so this needs a new wire contract for marginal benefit; the worst case is still a forced metadata COPY.
  • A distributed causal clock, or regenerating timestamps at commit. The wall-clock model plus the in-lock monotonic guard is the minimal correct fix.
  • Per-pool monotonic guards only. Ordinary reads return single-pool object info, so a sender could emit a stale primary-pool revision; the unified multi-pool value is required.

Verification and what it does not prove

R4 regressions cover the encryption × trust × timestamp matrix and the ordered sequence on two single-pool backends (single-drive and 16-drive erasure), with a stub KMS. R5 separately has a multi-pool tagging-deletion regression. The R4–R8 integration run repeated targeted tests on the merged tree, including the isolated R5 multi-pool test; all eleven PR #196 checks passed. These establish the tested ordering cases. They do not establish multi-site scheduling under real clock skew, cross-region failover, or behavior of deployments that upgrade one end of a pair only.

The upgrade summary and release boundary live in the component matrix; the sibling repair that stops normalized replica metadata from being re-injected is recorded separately in Replica Metadata Normalization.

Related records: tags · metadata · HTTP · audit

4.10 - SILO Server 20260903 Pre-release Review

This is the durable pre-release engineering record behind SILO 20260903. It explains why an earlier “all issues are solved” assessment was not accepted at face value, what the independent review found, how the fixes were narrowed, and which gates still remained at the review point. The linked release note records the later publication result.

Decision: the source candidate at 6e112d1856d4f3655f30fc81ee47e9f43d50d8f3 is a code-level GO for remote review. Production release remains a conditional GO until remote CI, Test Release, tag and artifact verification, signing, container publication, and public pull checks complete.
Baseline: RELEASE.2026-08-06T00-00-00Z at 3be10fcc1a44f6620ded0bd303461f9d688cca23.
Scope: SILO Server behavior and its embedded/pinned runtime components. Documentation, the standalone Console, mcli, package repositories, images, and the deployed site are separate deliverables.
Publication closure: the later final tree 9b11dc9469e650815b775cb47b039610644f5da4 was published as RELEASE.2026-09-03T13-18-01Z on 2026-09-04 after the remote, package, provenance, container, and public-download gates below completed. The conditional decision in this page remains the historical review criterion, not the current release state.

2026-09-09 follow-up: the repairs and main validation for #153, #152, and #137 are recorded separately in Replication Reliability. That record preserves #136’s accounting contract, explains cancellation lifecycle decisions, and keeps #133 separately tracked. It does not change the historical assessment of the 0903 release candidate below.

Why the second review was necessary

The first implementation pass had strong test results and resolved most reported defects. Its conclusion was nevertheless too broad: it treated green tests and a clean worktree as proof that every security invariant had been closed.

An adversarial review asked different questions:

  • Can the same invariant be bypassed by a different valid wire representation?
  • Does a pre-authentication fast path still perform I/O or acquire state?
  • What happens when metadata exists but cannot be loaded?
  • Do two individually correct read-modify-write paths share the same serialization boundary?
  • Does request sanitization preserve all SigV4 streaming state?
  • Does a validation claim describe the final tree or an earlier one?
  • Is a complex mechanism protecting a reproduced failure, or only a hypothetical future?

That pass found real defects after the initial “ready” claim. The correct response was not to distrust all prior work, but to narrow every assertion to an invariant and an observed tree.

Review result by area

Area Adversarial finding Final resolution Status
Bucket metadata Independent config locks could lose updates to the shared .metadata.bin record (#102) One bounded metadata.lock surrounds every whole-record writer, migration, import, adoption, and healing path; changed-field replication avoids stale whole-record replacement Closed in candidate
Bucket creation ForceCreate and site adoption could replace existing config with defaults Preserve existing records and update only creation/adoption state; add regression tests for clobbering Closed in candidate
Object Lock Comparing lock-document bytes with one canonical XML document missed valid configurations carrying a Default Retention rule Parse Object Lock first, then derive the versioning invariant from the parsed enabled state; verify update, read-back, and disk reload Closed in candidate
Pre-auth CORS Arbitrary path segments could cause metadata reads and cache growth CORS lookup reads resident metadata only and does no object-layer I/O Closed in candidate
CORS startup A nonresident name could fall back to global CORS before metadata initialization Preserve an explicit fail-closed startup state Closed in candidate
CORS load failure Forgetting that a real bucket failed to load made it indistinguishable from a nonexistent bucket and exposed the global fallback to pre-signed requests Maintain a bounded failed-bucket set, clear it on every successful load/remove/refresh path, and keep those buckets fail-closed Closed in candidate
CORS recovery A successful on-demand GetConfig reload did not initially clear the load-failure bit One-line final fix 84e1580a4 plus targeted race coverage Closed in candidate
Replication trust Presence of client-controlled internal headers enabled privileged behavior in multiple handlers Authenticate first; require an exact marker plus s3:ReplicateObject or s3:ReplicateDelete; carry a private context decision; sanitize untrusted headers afterward Closed in candidate
Streaming uploads The sanitized request clone did not initially share the original trailer map Preserve the trailer map so late-arriving streaming checksums remain visible Closed in candidate
Snowball A request-wide trust bit could leak between extracted entries Derive and isolate trust per entry; preserve request defaults across workers Closed in candidate
SSE-C Zero-byte reads and GetObjectAttributes could skip customer-key authentication Require a successfully unsealed key, with a separate authorized-replica exception Closed in candidate
Delete authorization Explicit version deletes checked the ordinary delete action instead of requiring s3:DeleteObjectVersion Align single and multi-delete authorization, keep replication deletes on s3:ReplicateDelete, and preserve auth/audit context Closed in candidate
Admin authorization User/group status changes always checked the enable action Check the action that matches the target state Closed in candidate
Checksums Multipart and copy paths omitted fields, accepted invalid combinations, or computed over the wrong representation Complete algorithm/type validation, server-side part calculation, federated propagation, AWS errors, and CopyObject transform ordering Closed in candidate
Release evidence Full acceptance initially described a tree that changed afterward Record full acceptance at ebac0ca73 and current-tree targeted gates separately Closed as an evidence defect

The invariants that now define the candidate

Trust is derived once, after authentication

An internal-looking header is still client input. The request must first pass the existing authentication path in its original signed form. Only then can the handler combine:

  1. an exact, single replication marker;
  2. a non-anonymous authenticated identity;
  3. s3:ReplicateObject or s3:ReplicateDelete on the addressed resource;
  4. replica status where the narrower replica-only semantics require it.

The result lives in private request context. Header stripping is defense in depth for legacy consumers, not the source of authority.

This ordering matters because SigV4 may sign the headers. Sanitizing first would reject legitimate replication with SignatureDoesNotMatch. The sanitized clone also has to share the request trailer: trailers arrive after the initial header parse and carry streaming checksums.

The complete receiver-wide model is in No I/O Before Auth, No Privilege From Headers.

A shared record has one write boundary

Policy, lifecycle, SSE, tags, quota, replication, Object Lock, versioning, and CORS are logical fields but physical members of one bucket record. A per-field mutex cannot protect a whole-record read-modify-write.

The selected repair is deliberately smaller than a new database or transaction layer:

acquire metadata.lock
  load or reuse current record
  mutate the requested field
  parse/normalize the complete record
  persist atomically
  publish the in-memory record
release metadata.lock

The lock does not cover object data I/O and is bounded to a bucket-metadata operation. Migration and healing must participate because they also replace the whole record. Replication receivers merge only changed fields so an older remote snapshot cannot erase unrelated local state.

Failure is a state, not the same thing as absence

The CORS hot path must distinguish four states:

State Result
Metadata system not initialized No CORS headers
Known real bucket whose metadata load failed No CORS headers
Resident bucket with a bucket CORS document Evaluate that document
No resident metadata and no known failure Use the server-wide fallback

The second row is why a failed-bucket set survives the simplification pass. A pre-signed URL is already authorized by its signature and may access a private object without bucket-policy evaluation. In that case the bucket CORS document is the browser-origin boundary. Losing the failure bit and using a permissive global fallback would weaken that boundary.

The set remains bounded by real bucket load attempts and is maintained through two helpers. Successful load, removal, stale-bucket cleanup, refresh, reset, and concurrent load all have tests.

Object Lock is semantic, not textual

Any valid enabled Object Lock configuration implies versioning. XML whitespace, element order, and the presence of a Default Retention rule do not change that meaning. Therefore normalization follows parsing, not a byte comparison against one canonical document.

The resulting versioning record is plain Enabled. A suspended state and an exclude-prefix extension are incompatible with the lock invariant and are removed on update, read-back, and reload.

Complexity audit

The pre-release pass explicitly looked for over-design, duplication, defensive programming without a threat model, and stale compatibility machinery.

Complexity retained because it protects a reproduced failure

  • One metadata lock: retained because a deterministic cross-type lost-update test reproduced data loss.
  • CORS tombstones: retained because site replication cannot distinguish deletion from “never observed” without them.
  • CORS load-failure state: retained because a pre-signed URL provides an authenticated, policy-independent counterexample.
  • Two replication trust levels: retained because ordinary replication and replica-ciphertext/SSE semantics do not use identical wire shapes.
  • Post-authentication sanitization: retained because sanitizing before SigV4 verification breaks legitimate signed requests.
  • Adversarial multi-pool/null-version tests: retained because single-pool happy paths do not exercise the state-selection failures they caught.

Complexity removed or narrowed

  • CORS failure-set mutations were centralized in noteLoadFailure and clearLoadFailure.
  • The replication import path now applies changed fields rather than copying an entire possibly stale record.
  • Obsolete encryption helpers, dead event-target functions, and abandoned handler branches were deleted.
  • The compatibility guard stopped inventorying every exported source symbol and now protects the actual served routes and frozen wire/configuration surfaces.
  • The old wait_pipe lint exemption was removed; gomodguard_v2 replaced deprecated configuration.
  • Dynamic timeout tests no longer call global rand.Seed from a parallel package.
  • The server returned from the temporary silo-go fork to the reviewed upstream-compatible minio-go revision.

Changes deliberately not introduced

  • no generic metadata transaction framework;
  • no second CORS cache or unbounded negative cache;
  • no new public “trusted replication” request header;
  • no cross-repository release gate that makes the server depend on a later Console or documentation release;
  • no partial conditional-delete contract in the release candidate;
  • no broad rewrite of inherited site-replication registers without dedicated convergence tests.

Deferrals and why they do not all have the same severity

Item Classification Release decision
Conditional delete #10 Inherited missing S3 feature; dangerous only to callers that assume unsupported If-Match / per-object ETag is enforced Document prominently; do not merge the incomplete PR or a single-only half contract
Multi-site config deletion #77 Inherited convergence defect for policy/SSE/tags/quota; CORS has its own fixed register Not a single-site blocker; deployment condition for users relying on those multi-site deletes
ListMultipartUploads #79 Inherited listing-conformance gap Known issue; not a data-integrity blocker for ordinary multipart workflows
Federated CopyObject #99, #100 Legacy-backend checksum/inline-object gaps Block use of the affected features, not the general server release
ILM relocation PR #60 and broad SSE issue #61 New capability requests Outside the release safety boundary

“Inherited” does not mean harmless. It means the defect was not introduced by this change set and should be evaluated against the documented release contract. A deployment that depends on one of the affected paths inherits a deployment-specific stop condition even when the general release remains conditional GO.

Evidence

Full acceptance tree

The full local acceptance corresponds to ebac0ca73bbf251b070bb6df4d8005015841f901:

  • full cmd and internal suites;
  • complete cmd race suite: 365.448 seconds, pass;
  • lint: 0 issues;
  • rebrand/compatibility and generated-file guards;
  • govulncheck with no reachable vulnerability;
  • six make verify deployment shapes: 174 PASS / 0 FAIL.

The first two make verify attempts encountered environment/setup failures while obtaining mcli, not test failures. The successful run used the locally checksum-pinned mcli, retained the outbound proxy for GitHub downloads, bypassed it for localhost, and placed GNU userland tools first in PATH. That distinction is part of the evidence rather than something to hide.

Post-acceptance candidate

The only code change after that full run is 84e1580a4, which clears one CORS failure-state bit after a successful on-demand metadata reload. The candidate merge adds no code; 6e112d185 changes only Helm release metadata and documentation. On the final candidate, the following pass:

  • git diff --check;
  • targeted CORS and Object Lock go test -race;
  • rebrand guard;
  • generated-file check;
  • lint with 0 issues.
  • Helm lint, default and optional renders, chart packaging, and the seven-resource legacy-upgrade identity guard.

This evidence is proportional to a one-line state-transition fix, but the remote CI and release workflows must still run against the pushed tree.

Go, no-go, and ownership of the remaining gates

Code decision: GO

No confirmed code defect from the two review rounds remains unresolved in the candidate. The fixes are covered at the layer where their invariants live, and the retained complexity corresponds to reproduced counterexamples.

Production decision: conditional GO

The server must not be described as released until all of these are facts:

  1. candidate commits are pushed and reviewed;
  2. remote CI and Test Release pass on the pushed head;
  3. the intended tag points at the reviewed chart 7.0.2/server 0903/client 0903 release tree;
  4. Draft artifacts, checksums, SBOMs, attestations, and signed RPMs verify;
  5. finalize and Docker release publish both classic and distroless variants;
  6. anonymous download and pull tests pass;
  7. release notes are updated from the tagged facts and the documentation site is deployed.

Any failure in steps 1–6 is a release blocker. A local green suite cannot substitute for them.

Deployment-specific stop conditions

Operators should delay even a successfully published release when they cannot yet:

  • update every node in a distributed cluster within one coordinated maintenance operation;
  • update every member of a site-replication group before using bucket CORS;
  • revise IAM policies for s3:DeleteObjectVersion and status-action separation;
  • avoid or explicitly accept the known #10, #77, #79, #99, or #100 path their workload depends on.

The final conclusion at the review point was intentionally narrower than “everything is fixed”: the reviewed candidate was ready to enter the release machinery, the remaining limitations were explicit, and production publication was gated by verifiable artifacts rather than confidence. Those gates later completed for the release linked above; these deployment conditions describe Server 20260903, the release this record reviewed.

Restart and readback verification (2026-09-11, #116)

A later acceptance (#116, run on 2026-09-11) measured bounded restart/readback behavior, including the inherited readiness gap. The durable part is the method, which any operator can reuse when validating a restart or upgrade window:

  • Acknowledgement ledger. Every acknowledged PUT writes to a unique versioned key, and the acknowledgement records the VersionId, byte count, and SHA-256 immediately. Readback fetches by exact VersionId and asserts all three — an early confirmation cannot be silently replaced by a later write.
  • Readback timing. Periodic re-reads happen at 15/30/60 s after the data canary succeeds, through every peer, and the final check includes writes made during a single-node outage and after its rejoin.
  • No retry masking. Each canary carries a hard 60 s deadline covering setup, request, response-body read, and sleep, and SDK retries are disabled so a recovery window cannot be papered over by client-side retries.
  • Driver topology. Four Linux/arm64 containers on one host with independent network identities, one drive each (EC 2+2), and tmpfs volumes kept mounted by a holder container through the full shutdown; nodes stop in parallel with a 10 s grace period, then start in parallel.

The operational finding worth remembering: admin-endpoint readiness is not data readiness. The fixed historical record tested 0806, 0903 and the then-current main repair build. The main build reached the admin gate 2.461 s after restart, then needed another 14.489 s for the data canary: different timing origins. The 0903 run took 0.191 s after its gate, which does not establish immunity to the inherited startup window. A readiness probe against admin/health says nothing about the data plane during that window, and no fixed sleep substitutes for an actual data-plane check.

Boundaries, stated as boundaries: this acceptance covers process/container restart and TCP peer reconnection on a single Linux host. It does not prove persistence across independent hosts, host reboots, or physical media failure, and the timings are individual observations, not latency guarantees. The run artifacts are retained outside the documentation tree; the method above is the part that generalizes.

September 16 source follow-up: #10 closed through the independent single-object repair #145; batch ETag deletion is still absent. #77 and #99/#100 were repaired on main. #133/#144 were subsequently addressed by #178; #79’s default listing limitations remain open. These changes must not be retroactively attributed to Server 20260903. See the current component/source matrix.

4.11 - Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject

This page records the repair of how a trusted replication receiver restores metadata for a replica, merged into Server main as PR #194 (fix 4fcdf37ce, merged as 9f3037e941).

As of 2026-09-16: the fix is on verified main 40220bd836cb. It is not in the published Server 20260903.
Provenance: the production logic follows PR #187 by Mikhail Khadarenka; the merged change keeps that authorship and narrows it to a review-validated boundary.
Evidence class: an HTTP-level baseline of 64 leaf cases (44 controls passing, 20 defect failures before the fix) against real single-drive and 16-drive erasure backends, plus counterfactual replay of the same suite against the baseline helper. No customer incident is attributed.

What went wrong

The ordinary PUT path normalizes metadata: it strips the transport-only aws-chunked token from Content-Encoding, and removes the X-Amz-Meta-X-Amz-Unencrypted-Content-Length/-Md5 user-metadata keys that a GHSA-76wf-9vgp-pj7w mitigation deliberately deletes. The trusted replication receiver, however, restored replica metadata by re-running the same permissive extractor with replication allowed — replaying every supported header and all user metadata from the original request. Concretely, on a trusted replica write the server could store and later return via GET/HEAD:

  • Content-Encoding: aws-chunked (a pure transport encoding that must never be stored, per the AWS SigV4 streaming rules), or the unsplit aws-chunked,gzip string instead of gzip;
  • the two GHSA-redacted user-metadata keys — a partial rollback of that mitigation, limited to trusted replica writes;
  • for Snowball entries without their own PAX header: the outer archive’s content-type, cache-control, and user metadata.

The object bytes themselves were not necessarily damaged; the stored metadata was wrong. The regression was introduced by 56fa63bfd (2026-04-15, the replication header trust boundary hardening, CVE-2026-34204) — whose trust protection is correct and stays.

The fix

One file (cmd/handler-utils.go). The boolean dual-mode helper is deleted:

  • the ordinary extractor unconditionally skips replication-only keys;
  • a new replica extractor walks only the replication-to-internal header map and restores only the six replication-scoped fields: the SSE-C sealed key material, sealed algorithm, IV, and encrypted-multipart marker (the empty marker is honored by key presence), the actual object size, and the SSE-C checksum identity mapping;
  • it never re-reads ordinary supported headers or user metadata.

Expected stored encodings after the fix:

Requested encoding Stored Content-Encoding
aws-chunked none
aws-chunked,gzip gzip
gzip gzip

aws-chunked, gzip (note the space) still stores gzip with a leading space, and gzip, aws-chunked stores the whole string. These are documented status quo, asserted by tests as such — not claims of repair.

Operator-visible changes

  • Trusted Snowball entries no longer inherit the outer archive’s ordinary metadata. Without PAX records, this removes outer content-type, cache-control, expires and user metadata; with PAX records it removes fields the entry did not restate. Per-entry minio.metadata.* still applies, the six replication-scoped fields still apply to authorized entries, and the archive’s storage class remains inherited. The in-tree batch producer calls PutObjectsSnowball, whose SDK emits the auto-extract marker, but it does not mark the outer request as a trusted replica; it did not use the affected inheritance path.
  • The GHSA-redacted keys are no longer written back on replica restore — matching what every ordinary PUT already did.
  • Authentication, permission gating, and replication trust semantics are unchanged; the ordinary extraction path is byte-for-byte equivalent.
  • Rolling back the code reopens the injection path but does not repair already-stored metadata.

Upgrade does not fix stored objects

An upgrade stops new pollution; it does not scan or rewrite existing objects. Two consequences matter:

  • Objects whose authoritative source remains polluted can be selected repeatedly for metadata replication when comparison with a normalized replica detects a difference. Fix the authoritative source first, then let the copies converge.
  • Ordinary S3 self-COPY is not a general remediation API: it can create new versions or shift timestamps rather than rewriting one version’s metadata in place.

The read-only audit runbook now provides an executable inventory tool and classification rules. It does not authorize or perform repairs.

The stored-metadata remediation proposal — status

A design for a future operation exists: build an inventory (including non-current versions, not just the latest), verify by comparing against the trusted source version or an independent checksum — never by guessing from the wrong response header, and never by re-decompressing gzip just because a label says so — process the authoritative source’s exact versions first, then converge copies; keep immutable inventories and metadata backups; use small validation batches with a rehearsed rollback. Where no supported path exists for an object, stop and leave it — editing xl.meta directly is not a supported operation.

Any selected procedure must preserve the required version identity and current version relationship, Object Lock retention/legal hold, tags, replication state, and encryption context. Check for concurrent changes before writing; blocked or unverifiable versions stay untouched. A local clone must prove the chosen operation and rollback before this proposal becomes an executable runbook.

This is a design proposal awaiting separate approval, not an executed procedure. No production inventory scan, object write, version change, or deployment has been performed as part of it. Treat it as the shape of a future runbook, not as a validated one.

Known limits

  • The POST-form upload path (bucket-handlers.go) calls the low-level extractor directly and never normalized encodings; that behavior is unchanged and flagged for a separate issue.
  • Local verification ran with a test-only capacity accommodation (a full host disk); the merged-tree rerun in the R4–R8 integration record covers the unchanged-tree case.
  • No two-site scheduler, restart, or network-failure acceptance is claimed.

Verification

Regression tests (TestExtractReplicationMetadata*, TestAPIReplicaContentEncoding, TestAPISnowballReplicaContentEncoding, plus race-included trust/SSE-C round-trips) cover the mapping table, the six restored fields, and the ordinary path’s equivalence; the recorded counterfactual run against the baseline helper produced 36 expected failures (20 HTTP cases and 16 helper cases), with 44 controls passing. These are the original repair’s observations, not a new execution by this documentation update. The upgrade summary lives in the component matrix; the sibling tag-ordering repair is recorded in Replicated Tag Ordering.

Related records: tags · metadata · HTTP · audit

4.12 - Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts

This page records the repair of Server’s HTTP read deadlines, merged into main as part of PR #196 (fix 055030ea5).

As of 2026-09-16: the fix is on verified main 40220bd836cb. It is not in the published Server 20260903.
Evidence class: synthetic — a direct TCP comparison (header limited to 100 ms, header arriving over 400 ms, standard Go refuses where old SILO returned 204), plus a real single-drive process probe where flag and environment settings both rejected a 400 ms slow header while healthy requests continued. No production incident is attributed; the issue that motivated the investigation is a slow-HTTP DoS scanner report that has not been reproduced against a deployed cluster.

Two timeout classes, one connection

  • Request-header absolute deadline. ReadHeaderTimeout bounds the total time from the start of header reading to its completion. Trickle-feeding bytes cannot extend it. On HTTP/1 keep-alive connections, Go first uses IdleTimeout while waiting for the next request’s initial bytes, then starts a fresh header deadline. ReadHeaderTimeout also participates in the separate TLS-handshake timeout calculation.
  • Body rolling idle timeout. Once headers parse and the connection enters the active phase, the existing rolling semantics return: every successful read extends the deadline, and only a stall between bytes (the configured idle timeout) kills the connection. A long upload or download that keeps making progress is not capped in total duration by this HTTP/1 repair; other protocol, proxy, and application timeouts still apply.

Before this repair, the first class did not exist in practice: the connection-layer wrapper replaced the socket deadline with now + idle + 250 ms before every partial read, overwriting whatever absolute deadline net/http had set — so a slow reader could hold a connection open indefinitely by sending one byte per idle window.

Two independent defects

  1. The connection layer neutralized the absolute deadline. The DeadlineConn wrapper’s read path reset the socket deadline on every partial read, defeating the read-header deadline Go’s server sets. The direct-TCP baseline proved it in isolation: with a 100 ms header limit and a 2 s idle window, a header that finishes at 400 ms was accepted.
  2. The configuration never reached the server. The CLI accepted --read-header-timeout and MINIO_READ_HEADER_TIMEOUT, parsed defaults and all — and the server-context builder copied IdleTimeout while dropping ReadHeaderTimeout entirely, so the running server always saw zero. Fixing defect 1 alone left the real process accepting slow headers; the second fix is one line next to the idle-timeout binding.

The reason this stayed invisible for so long: the flag’s default (30 s) equals the idle timeout’s default, and with the flag unwired the server fell back to exactly that same 30 s — so every observable default behaved as if configured.

Configuration

  • Flag: --read-header-timeout (Hidden: true, absent from ordinary CLI help)
  • Environment: MINIO_READ_HEADER_TIMEOUT
  • Default: 30 s (equal to the idle timeout default)
  • There is no YAML configuration field for either timeout; the value binds once at startup from flag > environment > default.
Setting Effect
header > 0 Absolute cap on HTTP/1 header phases; participates in Go’s TLS-handshake read window (including the HTTP/2 handshake)
header = 0 (explicit) Falls back to Go’s rule: the read timeout (= idle timeout) applies; the CLI default is 30 s
header < 0 Disables the header-specific cap. Positive read/write timeouts still bound TLS handshake reads, and positive IdleTimeout still bounds the keep-alive wait. This does not disable every connection timeout.
idle shortened, header unset Header phase independently uses the 30 s default — the one combination looser than a naive expectation, though still strictly tighter than the pre-fix unbounded extension

A negative value reopens unbounded slow-header trickling; it is not a recommended compatibility setting. An incomplete header cut off by the deadline normally sees a closed connection, not a guaranteed HTTP error status. The trigger was @AEGEGE’s scanner report #183; PR #195 was integrated through #196. The experiment does not establish reproduction in that deployment.

What each protocol gets

  • HTTP/1: headers and keep-alive waits are absolute; the body keeps the rolling idle timeout. The connection-state hook composes with (rather than replaces) any caller hook.
  • TLS: handshake reads take the minimum of the positive header deadline and the existing read/write timeouts; after the handshake, a fresh header limit begins. The handshake’s write side remains rolling — this repair is not a complete TLS-handshake resource limit.
  • HTTP/2: untouched. When h2 is negotiated, the phase switching is skipped entirely; h2 keeps its own native per-stream read timeout, which is absolute, and ReadHeaderTimeout never enters the h2 configuration.
  • Internal callers: Linux internode dialing uses its own rolling semantics; grid-hijacked connections unwrap to the raw TCP connection before any of this applies.

Rejected alternatives

  • Clamp all future deadlines globally. Go 1.27 sets a whole-request deadline in some paths; with the read timeout equal to the idle timeout, this would hard-cap entire HTTP/1 requests — header plus body — and kill every large upload.
  • Drop the read timeout and reinterpret zero as rolling idle. Zero is net/http’s “never time out” for background reads and hijacked connections; reinterpreting it would break long handlers, and h2 would lose its per-stream timeout.
  • Wrap the body reader / response controller. Full chunked/drain/EOF accounting with h2 special cases is a far larger change than the header defect requires. (A later, unmerged branch explores a body-side response controller for the same DoS family; as of this record it is not part of main and not part of this repair’s claims.)
  • Reconstruct the standard library’s deadline arithmetic in the hook. Duplicates stdlib internals that drift between Go versions; remembering the value stdlib actually asked for is the robust form.
  • Auto-derive strictness from value comparisons. With all three defaults equal at 30 s, “shorter than the idle window” is indistinguishable in production defaults; such logic only works in test configurations.

Verification and limits

Tests pin the connection wrapper across three consecutive update periods (no extrapolation of the absolute cap), the phase transitions over HTTP/1 keep-alive, TLS, and HTTP/2-only negotiation, and the flag/env binding on the real CLI context; a process probe exercised a live server with a 100 ms header limit rejecting a header that takes 400 ms. Known limits: the TLS handshake write side stays rolling; handler CPU/storage waits have no deadline; the absolute header cap carries no slack while the rolling idle keeps its ~250 ms update slack; and multi-node, cross-region long-transfer acceptance is future work — the integration record explicitly does not count a scripted S3 long transfer as passed for this repair.

The upgrade note (a shorter header timeout also narrows the TLS handshake window; it is not a total-duration limit for uploads or downloads) is in the component matrix.

Related records: tags · metadata · HTTP · audit

4.13 - Conditional DELETE: One Condition, One Logical Object

Status, 2026-09-16: PR #145 added single-object conditional deletion; PR #178 subsequently repaired multi-pool serialization and reconciliation. Neither change is in published Server 20260903. Check the component matrix before relying on this behavior.

The August proposal around PR #12 was broader than the code that merged. In particular, its proposed batch rejection, extra read authorization and current-version-only rule are not implemented guarantees. This page describes the maintained source at f99ed829b.

Implemented contract

Request Current main behavior
Single DeleteObject, nonempty If-Match: <ETag> Check the client-visible ETag under the deletion lock; mismatch returns 412 before deletion
If-Match: * Require an existing, non-delete-marker representation
Explicit versionId Evaluate the addressed version, not an unrelated current version
No If-Match, or an empty header value No conditional callback is installed; ordinary deletion applies
Batch DeleteObjects with per-item <ETag> The request model has no ETag field; the XML field supplies no deletion protection
Internal recursive x-minio-force-delete Prefix deletion returns before the object-condition path; do not use it as conditional deletion

These conditions do not add an s3:GetObject authorization check. Ordinary deletion authorization still applies, including s3:DeleteObjectVersion for an explicitly addressed version, Object Lock checks and the separate trusted-replication path.

Why the condition belongs above individual pools

An object can have copies of different ages in more than one pool. A request condition concerns the logical object selected for the operation. Evaluating and mutating independently in each pool can delete one copy and then return 412 from another, or delete the latest copy and expose an older one.

The multi-pool path therefore acquires the shared namespace write lock, gathers the relevant states, evaluates the condition once, clears the callback for lower layers, and reconciles the selected deletion across pools. The same lock boundary must cover concurrent writes and metadata updates. This is the purpose of the later multi-pool repair, not a claim that every physical disk is atomically updated.

Counterexamples from the early design

With old ETag A in one pool and current ETag B in another:

  • If-Match: A must not delete A first and only then fail against B.
  • If-Match: B must not remove B while leaving A to become visible again.

A per-pool HTTP callback also risks concurrent writes to one response writer. Consume the request condition at the coordinating layer instead.

Failure boundaries

The reconciliation path reads pool state before evaluating the condition and surfaces failures instead of treating an unknown pool as empty. Cleanup errors can still follow a partial physical mutation: a failed request is not a distributed rollback guarantee. Retrying and checking actual state remain necessary after a storage failure. See multi-pool consistency.

Selection and evaluation

Evaluate once

erasureServerPools.DeleteObject holds the outer lock. Multiple pools use deleteObjectReconciled; the single-pool path reads the selected representation and invokes CheckPrecondFn before its delete-marker shortcut. Lower layers do not reinterpret the condition per copy.

Wildcards and delete markers

The DELETE helper treats * as representation existence. A delete marker does not satisfy it. A missing key or addressed version follows the corresponding not-found path; it is not manufactured into an empty ETag match.

Authorization

The handler authorizes deletion using the effective version. It does not perform the additional s3:GetObject check described in the original proposal for a specific ETag. Do not treat the proposal’s permission matrix as implemented AWS parity.

SSE-C ETags

Conditional deletion compares the established client-visible ETag projection; it does not read or decrypt the object’s payload. SSE-C read-key authentication is a separate contract from deleting an object.

Explicit versions

An explicit versionId selects that version for comparison and deletion. A matching historical ETag can therefore allow deletion of the historical version even if the current version has another ETag. This differs from the early proposal’s current-version-only rule.

Unsupported edges

An empty If-Match installs no condition. The recursive prefix-delete extension bypasses object preconditions. Batch XML ETags are not recognized by ObjectToDelete; there is no request-wide NotImplemented guard. Callers needing compare-and-delete must use the supported single-object path with a nonempty condition and verify their selected release.

Alternatives and scope

Changing a shared ETag comparator cannot fix pool selection or mutation ordering. Aggregating errors after per-pool callbacks cannot undo deletions already performed. A new transaction framework is unnecessary for consuming one callback at the existing namespace lock, but batch execution and policy enforcement require separate implementation.

Evidence and release boundary

The source evidence is the handler, pool coordinator, and batch request model. Relevant coverage includes ETag mismatch, wildcard/delete markers, explicit versions, quorum failures and multiple pools. Earlier local review or test claims for a different implementation do not establish these missing guards. A merged implementation and a released, deployed binary remain separate facts.

Follow-up work

Batch conditions

Implementing per-item ETags requires parsing them, evaluating each logical object under the correct lock, preserving quiet mode and reporting each failed condition in the per-item response. Until then, a batch ETag is ignored and must not be used as a concurrency guard.

Policy enforcement

The maintained policy package does not define s3:if-match. Supporting it requires a package change and release, correct request condition values, a Server dependency update and authorization tests. Existing If-Match execution does not by itself provide policy enforcement.

Design cost

Single-object execution reuses the established object-selection and locking boundary. Full batch conditions and policy enforcement cross additional interfaces and remain separate work. The useful invariant is narrow: a false supported single-object condition must be decided before deletion, once for the selected logical representation.

4.14 - DSN-Only Database Notifications: A Compatibility Boundary for #53

Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.

This document is the product requirements and final design record for SILO issue #53. It records the accepted compatibility boundary, implementation, and verification for PostgreSQL and MySQL bucket-notification targets.

Decision

SILO will retain PostgreSQL and MySQL notification targets, but support exactly one current configuration form for each:

  • PostgreSQL requires a complete connection_string.
  • MySQL requires a complete dsn_string.

The old five-field form — host, port, username, password, and database — remains unsupported by the current KV configuration system. SILO will not re-register those keys and will not synthesize a DSN from them during legacy migration.

The legacy migration contract is deliberately narrow:

Legacy target Result
Disabled Ignore it; no target is emitted.
Enabled with a non-empty connection_string or dsn_string Migrate only the canonical connection-string key and the other registered target settings.
Enabled with only discrete connection fields Reject migration and abort server startup before the new configuration is activated, with an actionable error that names the subsystem and target but never prints a credential.

This is a configuration-boundary decision, not removal of the database-notification feature.

Status: implemented in f1ba68358 and included in Server 20260903.
Owner: SILO server repository.
Tracking: pgsty/silo#53.
Target: the next SILO patch release after implementation and verification.

Context

SILO inherited two generations of database-notification configuration from MinIO.

The pre-KV JSON configuration could describe a database connection either as a complete string or as five fields:

host
port
username
password
database

The current KV configuration exposes only the driver-native form:

notify_postgres  -> connection_string
notify_mysql     -> dsn_string

This direction is not new. MinIO deprecated the five discrete fields in RELEASE.2020-04-10T03-34-42Z and instructed operators to move to connection_string or dsn_string. SILO’s current help tables, environment-variable documentation, and examples already present the complete string as the supported interface.

SILO is a new community fork with an explicit migration step. Its compatibility contract prioritizes the S3 and Admin APIs, current MINIO_* settings, on-disk data, and current KV configuration. It does not need to perpetuate every pre-2020 configuration spelling when a supported canonical form has existed for years.

The defect

Before the fix, the legacy migration helpers, SetNotifyPostgres and SetNotifyMySQL, wrote both forms into the new KV configuration. Even when the old target already had a complete connection string, the helpers also emitted all five discrete keys, usually with empty values.

The new parser rejects those keys because neither DefaultPostgresKVS nor DefaultMySQLKVS registers them. Key validation checks key presence, not whether the corresponding value is empty. Both legacy source forms therefore fail:

old complete string -> canonical string + five empty unknown keys -> rejected
old discrete fields -> empty canonical string + five populated unknown keys -> rejected

The failure is amplified by notification initialization. FetchEnabledTargets is fail-fast across notification subsystems: the first invalid subsystem returns an error and a nil target list. The caller logs the error and continues starting the object server, leaving healthy Webhook, Kafka, NATS, and other targets unavailable as well.

Merely returning an error from the two migration helpers does not fix that behavior. The error propagates through readConfigWithoutMigrate and initConfig, but initConfigSubsystem currently logs non-retriable configuration errors as “some features may be missing” and returns success. The server then starts without assigning globalServerConfig; notification failure is only one consequence, because region, storage class, compression, identity, and other stored settings may also be absent. The implementation must therefore carry a typed database-migration error to the startup boundary and make that error fatal. Classifying it as retriable is also wrong because the server would retry forever without any state change that could repair the configuration.

The resulting behavior is especially dangerous because object I/O still works. Operators can see a healthy S3 service while every configured event pipeline has stopped. Targets are never constructed, so delivery or later replay of events produced during the outage must not be assumed.

There is also a diagnostic-exposure issue. The unregistered password key has no sensitivity metadata and may be copied verbatim into health or diagnostic material. The registered connection_string and dsn_string keys are already treated as sensitive values.

Why the first fix was reverted

The first repair registered the five discrete keys and taught the parser to read them. That made migrated targets pass CheckValidKeys, and it appeared attractive because the target argument structures and constructors still contain code for the old fields.

It also broke the documented connection-string path.

The shared mc admin config set tokenizer discovers field boundaries by looking for registered key names. It is not fully quote-aware. Once port became a registered key, this valid input contained what looked like a second top-level field:

connection_string="host=db port=5432 dbname=events user=app"

The tokenizer split at the port= inside the quoted value, truncated connection_string, and handed the remainder to the port parser. The command then failed with invalid port.

Under the current tokenizer, registering common words such as host, port, and password creates a direct conflict between the connection-string grammar and the top-level KV grammar. The attempted registration fix was therefore reverted. Re-registering those keys is not an acceptable solution.

Product judgment

Database notification targets are a specialized but useful capability. They provide a direct database-backed namespace view or access journal without requiring an external event bus. That remains valuable for small deployments and for users already operating PostgreSQL or MySQL.

The legacy spelling of their connection parameters has much less value. A five-field model cannot represent the useful range of driver options: TLS modes and certificates, connection timeouts, application names, Unix sockets, multi-host PostgreSQL settings, MySQL driver parameters, and future driver capabilities. Supporting both forms also creates precedence, merging, redaction, and testing questions that do not exist with one canonical value.

The complete string is the better abstraction boundary: SILO owns notification semantics, while the database driver owns connection syntax.

The product decision is therefore to keep the capability and remove the compatibility illusion. An unsupported legacy target must be rejected clearly; it must not be accepted and transformed into a configuration that later disables unrelated targets.

Goals

  1. Establish connection_string and dsn_string as the only supported live configuration interfaces for database notifications.
  2. Allow a legacy JSON target that already contains the canonical string to cross the migration boundary without modification to its connection semantics.
  3. Reject enabled discrete-only legacy targets before a partial or invalid KV configuration is activated.
  4. Replace the current silent runtime failure mode of #53 — healthy targets disabled while the server appears healthy — with an explicit startup-time failure that operators must resolve before the server runs.
  5. Ensure no migration error, log line, health report, or diagnostic bundle exposes a database password.
  6. Remove the ten Postgres/MySQL exceptions from the source-level unregistered-write audit.
  7. Make the compatibility boundary and operator remediation explicit in release and migration documentation.

Non-goals

  • Supporting both DSN and discrete database fields in the current KV interface.
  • Automatically synthesizing a DSN from old discrete fields.
  • Rewriting the shared KV tokenizer.
  • Changing FetchEnabledTargets fail-fast semantics in this patch.
  • Silently skipping an enabled database target and continuing with partial notification coverage.
  • Removing PostgreSQL or MySQL notification targets.
  • Deleting the legacy struct fields needed to decode and identify unsupported input. They remain on shared target argument structs that are also used by live constructors, whose discrete-field connection-string synthesis is unreachable from current KV configuration; those fields must not become supported configuration keys.
  • Correcting ignored errors from the other eight legacy notification setters. Their pre-existing silent-skip behavior remains unchanged in this narrowly scoped database-migration patch and requires a separate audit and design decision.

Functional requirements

Current configuration

  1. notify_postgres accepts connection_string; notify_mysql accepts dsn_string.
  2. The five discrete keys remain unregistered and rejected by current configuration commands.
  3. Existing full strings must continue to support the database driver’s syntax, including parameters whose names contain host, port, user, password, or database.
  4. No new public environment variables or KV keys are introduced.
  5. The declared legacy variables MINIO_NOTIFY_POSTGRES_HOST/PORT/USERNAME/PASSWORD/DATABASE and their MySQL equivalents are not wired into current parsing and remain unsupported. They must not be documented as working alternatives to the complete-string variables.

Legacy migration

  1. SetNotifyPostgres must return without emitting a target when the legacy target is disabled.
  2. For an enabled target, SetNotifyPostgres must require a non-empty ConnectionString and write only registered Postgres keys. If both a canonical string and discrete fields are present, the canonical string wins and every discrete value is discarded.
  3. SetNotifyMySQL must apply the equivalent rule to DSN.
  4. Neither helper may emit host, port, username, password, or database.
  5. A missing canonical string must return a typed or wrapped migration error identifying the subsystem and target name.
  6. cmd/config-migrate.go must check and propagate both helper errors. Ignoring them is forbidden.
  7. No partially migrated configuration may be activated or persisted after either helper fails.
  8. Error text may name the required key and remediation, but must not include any connection-field value.
  9. The propagated typed migration error must abort server startup. It must not be downgraded to the non-fatal “some features may be missing” path in initConfigSubsystem, and it must not enter the retriable-error loop.
  10. Validation errors for a supplied canonical string follow the same startup-fatal and secrecy rules; wrapping must add target context without repeating the DSN or its components.

Original proposed error shape (illustrative, not the shipped literal):

notify_postgres:archive uses unsupported legacy discrete connection fields;
set connection_string before migrating to SILO

Operator remediation

An operator encountering the error must choose an explicit remediation path. This applies both before an initial switch to SILO and when upgrading a deployment that is already running SILO: legacy migration output is not persisted, so the same old JSON source can re-enter migration on every start. A deployment that currently starts with notifications silently broken can therefore fail to start after this repair until the source configuration is corrected.

  1. On a compatible intermediate MinIO release, replace the old fields with connection_string or dsn_string, verify the target, and then migrate to SILO.
  2. Disable or remove the legacy database target, migrate the server, and recreate the target with the canonical string afterward.
  3. For a fresh SILO installation, create the target directly with the canonical string; no legacy migration is involved.
  4. For an existing SILO deployment that still reads a legacy JSON file, stop on the previous working release, back up the source configuration, then convert, disable, or remove the database target before starting the fixed release. Do not delete or rewrite unrelated configuration.

Documentation must not suggest that a discrete-only target will be converted automatically.

Availability trade-off

This decision intentionally turns one unsupported configuration from a degraded startup into a hard startup failure. The immediate availability cost is real: a server that previously served objects while all notifications were silently dead may refuse to start after the repair.

That cost is accepted because an object server that appears healthy while configured event sinks are absent creates silent, potentially unrecoverable downstream data loss. SILO is a new fork with an explicit migration boundary, and the discrete form has been deprecated since 2020. A fatal, actionable precondition is preferable to an upgrade that reports success with reduced notification coverage. The release note must make this startup behavior prominent; it must not be buried as an internal migration cleanup.

Security requirements

  1. The unsupported-input error must never format the legacy argument structure or its values.
  2. Tests must use a sentinel password and assert that it is absent from returned errors and captured logs.
  3. Migrated output must contain the registered sensitive connection-string key and no standalone password key.
  4. If a diagnostic bundle was exported from an affected deployment before this repair, operators should treat the database password as potentially disclosed and rotate it.

Alternatives considered

Register and parse the discrete fields

Benefit: preserves the old source form and uses already existing argument fields.
Rejected because: registration makes common field names visible to the shared tokenizer and corrupts quoted connection strings. It also expands the supported public configuration surface after the fields were deprecated in 2020.

Synthesize a canonical string during migration

Benefit: preserves discrete-only legacy installations.
Rejected because: it creates permanent code and test ownership for an obsolete input form, including PostgreSQL quoting, MySQL DSN formatting, socket and IPv6 behavior, defaults, and future driver drift. For a new fork with an explicit migration boundary, the benefit does not justify the continuing surface.

Skip only the unsupported target

Benefit: keeps the object server and other notification targets running.
Rejected because: silently discarding a configured event sink can cause unobservable and unrecoverable event loss. A clear migration failure is safer than an apparently successful upgrade with reduced notification coverage.

Change global notification fail-fast behavior

Benefit: limits the blast radius of future invalid targets.
Rejected for this change because: it neither repairs the database target nor closes the credential-exposure path, and it changes system-wide error semantics. It may be evaluated independently with its own operational contract.

Remove database notification targets

Benefit: removes the complete database-specific maintenance surface.
Rejected because: the targets remain useful and self-contained. The defect belongs to an obsolete configuration form, not to the notification capability itself.

Implementation scope

The server change should remain narrow:

  1. Update internal/config/notify/legacy.go so the two database setters emit only canonical registered keys and reject enabled targets without a canonical string.
  2. Update cmd/config-migrate.go to propagate the two database-helper errors with subsystem and target context.
  3. Define a typed database-migration error and update cmd/server-main.go so initConfigSubsystem returns it as fatal instead of logging and ignoring it. It must remain non-retriable.
  4. Leave ignored errors from the other eight legacy notification setters unchanged in this patch; record them for a separate audit rather than expanding #53 implicitly.
  5. Remove all ten Postgres/MySQL entries from knownUnregisteredWrites; the ratchet should become empty unless another independently justified legacy exception exists.
  6. Add focused migration, startup, validation, secrecy, and coexistence tests.
  7. Update database-notification and migration documentation in silo.pgsty.com.

The patch must not register the old keys, change the generic tokenizer, or refactor unrelated notification targets.

Acceptance criteria

The implementation is complete only when all of the following are demonstrated:

  1. A legacy PostgreSQL target with a complete connection string migrates, passes CheckValidKeys, and is returned by GetNotifyPostgres unchanged.

  2. A legacy MySQL target with a complete DSN does the equivalent.

  3. Discrete-only enabled targets for both databases fail before target initialization with an actionable error containing the subsystem and target name, and server startup aborts.

  4. Missing-string and malformed-string errors contain none of the sentinel host, username, password, database, or DSN values.

  5. Disabled discrete legacy targets do not create configuration entries and do not block migration.

  6. Migrated KVS output contains none of the ten discrete keys, including empty ones.

  7. When a legacy target contains both a canonical string and conflicting discrete values, only the canonical string is migrated and no discrete sentinel appears in any output KVS value.

  8. A SetKVS regression test using the real DefaultPostgresKVS and DefaultMySQLKVS key sets accepts a quoted connection string containing port=, host=, or password=.

  9. A configuration containing healthy Webhook, Kafka, or NATS targets cannot reach FetchEnabledTargets with an invalid migrated database target because readConfigWithoutMigrate fails without yielding, persisting, or activating a partial configuration, and startup aborts on that typed error.

  10. initConfigSubsystem returns the typed migration error; it neither logs-and-continues nor enters the retriable loop.

  11. knownUnregisteredWrites no longer contains Postgres or MySQL exceptions.

  12. The following verification passes:

    go test ./internal/config/notify ./internal/config ./internal/event/target -count=1
    go test -v ./cmd -run 'Test(ReadConfigWithoutMigrate|InitConfigSubsystem)' -count=1
    git diff --check

    The verbose cmd output must show that tests with both prefixes actually ran; a zero-match warning is a failed acceptance check. The normal server CI suite must also pass. In the documentation checkout, run make check.

Implementation result

Server commit f1ba68358 implements the accepted design without expanding the public configuration surface:

  • the two legacy database setters emit only connection_string or dsn_string plus registered target settings;
  • disabled targets remain ignored, while enabled targets without a canonical string return a value-free LegacyDatabaseTargetError;
  • only the two database migration errors are newly propagated;
  • the typed error is non-retriable, escapes initConfigSubsystem, and is classified as fatal by serverMain before logger.FatalIf exits the process;
  • the ten Postgres/MySQL exceptions were removed from knownUnregisteredWrites;
  • focused tests cover complete-string round trips, canonical precedence, discarded discrete values, secrecy, failed-migration atomicity, startup classification, and the real tokenizer key sets.

The final local Claude Code review used Claude Fable 5 at max effort and returned GO with high confidence and no blocking findings. Verification included the focused package set, race tests, go vet ./cmd, and the complete go test ./cmd -count=1 suite. The review authorized only the six-file server commit; publication remains a separate gate.

Cross-repository review found no implementation changes are required in pgsty/mc, pgsty/silo-pkg, or pgsty/silo-console: the client forwards configuration text, the package repository owns no notification schema, and Console already serializes its form into the canonical connection_string or dsn_string. The public reference and compatibility documentation is updated with this record.

Release and compatibility statement

The release note must describe this as an enforced compatibility boundary:

SILO database notification targets require connection_string for PostgreSQL and dsn_string for MySQL. The pre-2020 discrete host/port/username/password/database form is not migrated. Convert or recreate such targets before switching the deployment to SILO.

Deployments already running SILO with an old-format source configuration are equally affected: after this release the server will not start until each enabled legacy database target is converted, disabled, or removed.

The issue should close only after the repair is present in a published server tag. A merged patch, a local site build, and a published release are separate completion gates.

Review record

Claude Fable 5 reviewed the first draft at xhigh effort on 2026-08-23 and returned approve with required changes. The required calibration was incorporated: startup-fatal propagation now extends through initConfigSubsystem; already-running SILO deployments are covered; the availability trade-off is explicit; canonical-string precedence, dead legacy environment variables, other ignored helper errors, and executable tests are specified.

The same model then completed a final source-backed verification pass. Final verdict: approve, with no blocking findings. It confirmed that the English and Chinese records are aligned, the requirements are implementable against the current server tree, and the acceptance criteria cover the startup, migration, parser-regression, and secrecy boundaries.

After implementation, a separate local Claude Code review using Claude Fable 5 at max effort traced the path through ExitFunc(1), inspected driver error behavior, ran the focused, race, vet, and full cmd suites, and returned GO with high confidence and no blocking findings.

4.15 - Preview Text, Never Execute It: SILO Console Text Preview PRD

Status: shipped in SILO Console 2.2.0 · Owner: pgsty/silo-console · Tracking: pgsty/silo#17 · Review: consensus of product, security, and frontend architecture reviews

SILO Console can preview images, PDFs, audio, and video, but not the small logs, text files, JSON documents, and XML documents that operators inspect every day. A correctly stored Content-Type does not help: these objects are classified as unsupported before the preview renderer is selected.

Restoring the old browser-native behavior would be easy. It would also be the wrong fix. An object in storage is controlled by the user who uploaded it. Loading that object as a same-origin HTML or XML document would turn a convenience feature into an execution boundary.

The accepted design therefore makes a stronger promise:

SILO previews eligible objects as bounded UTF-8 text. It never asks the browser to interpret their markup, MIME type, or file contents as a document.

This record fixes the product boundary, the resource limit, the security invariants, the implementation shape, and the evidence required before the feature can ship.

Decision

The first release will add a dedicated text preview type and a PreviewText component.

The contract is:

  1. Preserve every existing image, PDF, audio, and video classification.
  2. Only when the existing classifier returns none, consider a text fallback.
  3. Admit the four target extensions or four exact passive text MIME types.
  4. Fetch bytes through the ordinary authenticated download path, without preview=true.
  5. Enforce a hard application read limit of 1 MiB.
  6. Decode only strict UTF-8 and reject binary-looking content.
  7. Render one React text node inside a scrollable <pre>.
  8. Never use an iframe, HTML parser, XML parser, or HTML injection API.
  9. Show the complete object or no object; do not show a truncated JSON or XML document.
  10. Keep download available for files that are too large, invalidly encoded, or otherwise unavailable.

No new API route or S3 operation is added, and the backend inline MIME allowlist is unchanged. Delivery did change Console download responses: zero-byte Range requests return an empty 200, unsatisfiable ranges return 416 with Content-Range: bytes */N, and size is always emitted in object JSON. See Console 2.2.0.

Current behavior

The defect is present in SILO Console v2.1.1, the version currently pinned by SILO when this design was written.

The frontend preview union contains only:

image | pdf | audio | video | none

Its extension table contains media formats, but not .log, .txt, .json, or .xml. Its MIME classifier likewise ignores text/plain, application/json, application/xml, and text/xml.

Runtime verification produced this split:

Object Frontend result Console download response
.log / text/plain none inline, SAMEORIGIN
.txt / text/plain none inline, SAMEORIGIN
.json / application/json “Preview unavailable” inline, SAMEORIGIN
.xml / application/xml none attachment, DENY

The object-detail action also uses the wrong conjunction when deciding whether Preview should be disabled. An authorized user can click Preview for an unsupported object and receive only the unavailable message; in other combinations, the UI can offer an action before the server rejects it.

The preview component still contains a generic same-origin iframe fallback. It is unreachable under the current type union, so the current defect is not an exploitable text-preview XSS. The dead branch is nevertheless hazardous: adding text to the union and letting it fall through would reactivate precisely the document-loading behavior this design rejects.

Root cause

This is contract drift across three independently evolved layers.

Classification drift

The browser code decides eligibility from filename and object metadata, but its closed type union has no text representation. Correct metadata cannot select a renderer that does not exist.

Response-policy drift

The Console server separately decides whether a response may be inline. It still treats plain text and JSON as safe passive MIME types, while XML and HTML remain attachments. That server decision is not reflected in the frontend classifier.

Renderer drift

The old generic iframe remains after the set of reachable preview types became media-only. The code therefore suggests a capability that the type system can no longer invoke.

The repair must realign the three layers without making MIME metadata a security boundary.

Why same-origin iframe preview is rejected

X-Frame-Options: SAMEORIGIN is not a sandbox. It controls who may embed a response; it does not limit what code inside a same-origin frame can do.

If uploader-controlled HTML, XHTML, SVG, or active XML were ever served as an inline same-origin document, it could act with the Console origin. An HttpOnly cookie would prevent direct cookie reads, but it would not prevent authenticated same-origin requests. A permissive or accidentally widened MIME rule would then turn stored content into stored application code.

nosniff, Content Security Policy, and Content-Disposition remain useful defense in depth, but none replaces the core invariant:

untrusted object bytes
        |
        v
strict text decoder
        |
        v
React textContent

never:
iframe / innerHTML / DOMParser / XML parser / executable document

Product contract

The feature is a read-only text viewer, not a web previewer and not an online editor.

The user should be able to:

  • open a small eligible object from either the list or object-detail surface;
  • read whitespace-preserving source text in the existing preview modal;
  • select and copy text using browser-native behavior;
  • understand whether a failure is caused by size, encoding, permission, object replacement, or network error;
  • download the original bytes at any time.

The user must never be led to believe that:

  • formatted JSON is the stored object;
  • a partial XML document is complete;
  • replacement characters are original bytes;
  • an unsupported encoding has been decoded faithfully;
  • an active HTML/XML document has been safely “sanitized” and executed.

Goals and non-goals

Goals

  1. Preview small logs, text, JSON, and XML without a local download.
  2. Keep object content inert regardless of extension, MIME, or payload.
  3. Bound retained response bytes and rendered text to 1 MiB.
  4. Preserve the stored text rather than silently reformatting it.
  5. Keep list and detail actions consistent with permissions and type eligibility.
  6. Support current object versions and explicitly selected historical versions.
  7. Preserve anonymous-access and subpath-hosting behavior.
  8. Ship the feature in Console first, then consume that exact Console revision in SILO.

Non-goals

  • HTML or XHTML rendering.
  • XML parsing, XSLT, external entities, or schema validation.
  • Markdown rendering.
  • JSON pretty-printing.
  • YAML or CSV-specific behavior.
  • Editing or saving.
  • Syntax highlighting, line numbers, search, folding, ANSI rendering, or linkification.
  • Head, tail, or truncated previews for large objects.
  • Lossy decoding or automatic detection of GBK, UTF-16, Latin-1, or other encodings.
  • A new backend text-preview endpoint.
  • Changes to the existing SVG, media, PDF, download, share, or storage contracts.

An object such as notes.md may still be shown as raw text when its exact MIME type is text/plain. It does not gain Markdown semantics.

Eligibility contract

Eligibility is deliberately two-stage.

Stage 1: preserve the legacy media decision

Run the current image, PDF, audio, and video classifier unchanged. If it returns anything other than none, return that result.

This preserves historical behavior for conflicting filename and MIME combinations.

Stage 2: apply text fallback

Only after the legacy result is none:

  1. Reject final extensions .html, .htm, and .xhtml.

  2. Match the final filename extension case-insensitively against:

    • .log
    • .txt
    • .json
    • .xml
  3. Normalize Content-Type by removing parameters, trimming whitespace, and lowercasing it.

  4. Match the normalized MIME exactly against:

    • text/plain
    • application/json
    • application/xml
    • text/xml

An allowed extension or an allowed exact MIME is sufficient. Broad matches such as text/, substring tests, and application/+json are forbidden in this release.

The resulting matrix is normative:

Filename and MIME Result Reason
report.txt + image/png image Existing media decision wins.
report.json + application/pdf PDF Existing media decision wins.
server.LOG + application/octet-stream text Allowed extension, case-insensitive.
no extension + application/json; charset=utf-8 text Exact normalized MIME.
page.html + text/plain none Explicit active-extension exclusion.
page.txt + text/html text Extension admits it; HTML source remains inert text.
notes.md + text/plain text Exact MIME admits raw text, not Markdown rendering.
image.svg + image/svg+xml existing image path No new text or iframe path.

Filename and MIME affect product eligibility only. They never select an executable rendering mode.

Resource contract

The binary limit is:

MAX_TEXT_PREVIEW_BYTES = 1,048,576

Exactly 1 MiB is eligible. 1 MiB plus one byte is not.

Known sizes

  • If the selected version has a known size greater than the limit, the initial attempt does not request its body. Explicit Retry bypasses that possibly stale listed size but retains the bounded Range and byte limit.
  • A known zero size still takes the bounded request path; an empty response produces the empty-file state.
  • If its known size is within the limit, begin a bounded request.
  • An absent size is not the same as zero; it enters the bounded unknown-size path.

The current list-to-modal handoff must therefore preserve undefined rather than converting it to zero with a truthy fallback.

Bounded request

For a small or unknown size, request:

Range: bytes=0-1048576

The extra byte is an over-limit sentinel.

The client must:

  1. Inspect Content-Range and Content-Length when present.
  2. Read the response as a stream rather than calling response.text() or building a complete Blob.
  3. Retain at most the limit plus the sentinel byte.
  4. Cancel immediately when the sentinel byte is observed.
  5. Enforce the same limit when the server ignores Range and returns 200.
  6. Render only after end-of-stream proves that the complete object is within the limit.

An over-limit object opens an explanation state with its known size, the 1 MiB policy, and a Download action. It never shows a prefix fragment.

Request identity and cancellation

A preview request is identified by:

bucket + object name + version ID

The request must use the existing generated API client or an equivalent base-path-safe helper so that it preserves:

  • same-origin credentials;
  • the current Console subpath;
  • version_id;
  • anonymous-mode X-Anonymous: 1;
  • current error handling and permission boundaries.

Close, object change, version change, bucket change, and component unmount must abort the active request and clear the old content.

Abort alone is insufficient. A generation token or invalidation flag must also prevent a response that already completed reading or decoding from updating a newer preview.

An aborted request is not an error and must not produce an error toast.

Encoding and fidelity

The first release supports strict UTF-8 only:

new TextDecoder("utf-8", { fatal: true })

Requirements:

  • handle the UTF-8 BOM without displaying it;
  • preserve Unicode text, emoji, tabs, LF, and CRLF;
  • reject invalid UTF-8 rather than inserting replacement characters;
  • reject decoded NUL characters as binary or unsupported content;
  • do not guess another encoding;
  • do not log or persist object text;
  • always retain Download as the original-byte escape hatch.

The unsupported-encoding state should explain:

This object is not valid UTF-8 text or contains binary data. Download it to inspect the original bytes.

JSON and XML are displayed exactly as decoded source text. The first release must not run JSON.parse followed by JSON.stringify: that can alter unsafe integers, duplicate keys, whitespace, lexical forms, and the text users copy.

Safe renderer

The success state renders one text node:

<pre>{content}</pre>

The implementation must not use:

  • iframe, object, or embed;
  • dangerouslySetInnerHTML or innerHTML;
  • DOMParser or an XML parser;
  • Markdown or HTML rendering;
  • an HTML data/blob URL;
  • per-line or per-token spans;
  • automatic links, ANSI escapes, or syntax markup.

One bounded text node keeps the DOM cost predictable and the security property inspectable.

The preformatted region uses a monospace font, preserves whitespace, defaults to no wrapping, owns both scrollbars, is keyboard focusable, and supports native selection and copy. No-wrap is intentional: it preserves aligned logs and avoids expensive layout of a single very long line.

UI states and permissions

The Preview action is enabled only when:

eligible preview type
AND object read permission
AND not a delete marker
AND not a prefix

The object-detail conjunction bug must be fixed, and list and detail surfaces must share the same eligibility function.

An eligible over-limit object still offers Preview. The modal explains why content is not loaded; disabling the button would leave the user unable to distinguish size, permission, and type failures.

The modal distinguishes:

State Required behavior
Loading Accessible busy state; no stale text.
Success Scrollable raw text plus Download.
Empty Explicit “File is empty” state.
Too large Object size, 1 MiB limit, Download and Retry; no initial body request when size is known to exceed the limit.
Invalid UTF-8 / binary Dedicated explanation and Download.
Forbidden Permission-specific message; no retained text.
Not found / replaced Object-change message; no retained text.
Network / server error Actionable retry/download state.
Aborted / closed Silent cleanup.

HTTP error bodies must never be decoded and displayed as object content.

All new user-facing strings go through the existing translation layer and ship in English and Chinese together. The content region and controls must remain usable in light and dark themes and at narrow widths.

Functional and security requirements

Functional requirements

  • FR1: Existing media and PDF classification remains unchanged.
  • FR2: The text fallback follows the normative extension/MIME matrix.
  • FR3: Eligible complete objects up to 1 MiB render as strict UTF-8 source.
  • FR4: Over-limit objects render no partial content.
  • FR5: Empty objects have a distinct successful empty state.
  • FR6: Current and selected historical versions use the same version for metadata, size, and body.
  • FR7: Anonymous access and subpath hosting retain their current request behavior.
  • FR8: List and detail actions apply the same type and permission decision.
  • FR9: Download, share, media, PDF, and storage behavior do not change.

Security requirements

  • SR1: Object bytes can reach the DOM only through text content.
  • SR2: Text Preview contains no document renderer or parser.
  • SR3: At most 1 MiB plus one sentinel byte is retained.
  • SR4: Closing or changing identity invalidates every previous response.
  • SR5: Invalid UTF-8 and NUL content are not shown as faithful text.
  • SR6: Errors, Redux, local storage, logs, and telemetry never retain preview text.
  • SR7: Server authorization remains authoritative for direct requests.
  • SR8: No CSP or backend inline MIME relaxation is introduced.

Implementation scope

Expected Console changes:

  1. Refactor preview classification so the current media decision is preserved and text is an explicit fallback.
  2. Add text to the preview type union.
  3. Add a dedicated PreviewText component with streaming bounds, strict decode, request cancellation, and explicit states.
  4. Route text objects explicitly to that component.
  5. Remove the unreachable generic iframe fallback.
  6. Fix the object-detail Preview disable expression and share eligibility logic with the list surface.
  7. Preserve unknown size instead of coercing it to zero.
  8. Add English and Chinese strings.
  9. Add classification, component, resource, security, permission, version, and browser tests.

Expected unchanged areas:

  • Console and S3 API paths;
  • the backend safeMimeTypes list;
  • Content Security Policy;
  • object storage and metadata formats;
  • image, PDF, audio, video, download, and share handlers;
  • external frontend dependencies.

If a future product requires tailing, server-side transcoding, organization-wide policy, or reliable behavior through proxies that ignore Range, a dedicated server endpoint may be designed separately.

Rejected alternatives

Keep text preview disabled

Benefit: no new code or browser memory use.
Rejected because: logs and configuration objects are a routine object-storage workflow, and download-only inspection is an avoidable Console regression.

Reuse the same-origin iframe

Benefit: minimal code and browser-native presentation.
Rejected because: it turns uploader-controlled content and mutable MIME metadata into a same-origin document boundary. It also leaves resource use unbounded.

Add a backend preview API now

Benefit: central server-side limits and normalized text responses.
Rejected for the first release because: the user already has object-read permission, and the existing download endpoint provides versioning, authorization, and Range. A new API would duplicate contracts without establishing a new data-access boundary.

Show the first 1 MiB of a large object

Benefit: better large-log convenience.
Rejected because: partial JSON/XML is structurally misleading, UTF-8 boundaries need additional handling, and a single “preview” action would no longer mean complete content.

Decode invalid UTF-8 with replacement characters

Benefit: some damaged or legacy logs remain partially readable.
Rejected because: copied text would no longer faithfully represent the stored object. Lossy viewing and other encodings require a separate, explicit product mode.

Auto-format JSON

Benefit: more readable indentation.
Rejected because: parse/stringify can alter numbers, duplicate keys, lexical representation, and copied content. A future opt-in formatted view may sit beside, never replace, the raw default.

Add Monaco or another code editor

Benefit: line numbers, search, highlighting, and folding.
Rejected because: bundle, worker, CSP, and maintenance costs exceed the needs of a bounded read-only preview. A native <pre> is smaller and easier to audit.

Acceptance and test plan

Classification matrix

Automated tests must lock every normative matrix row, extension case handling, MIME parameter stripping, explicit HTML/XHTML denial, and unchanged media conflicts.

Resource tests

Cover:

  • 0 bytes;
  • 1 byte;
  • exactly 1,048,576 bytes;
  • 1,048,577 bytes;
  • known over-limit size with zero body requests;
  • unknown size;
  • 206 with a revealing Content-Range;
  • server ignores Range and returns 200;
  • missing or false Content-Length;
  • close and identity changes during streaming.

No case may retain or render more than the complete allowed object.

Encoding and fidelity tests

Cover UTF-8 Chinese, emoji, tabs, LF, CRLF, BOM, invalid byte sequences, NUL bytes, JSON unsafe integers, duplicate keys, original whitespace, XML declarations, DOCTYPE, CDATA, and stylesheet processing instructions.

The raw success view must preserve decoded text. Invalid and binary cases must show their dedicated state.

Security tests

Payloads containing <script>, event attributes, iframe tags, SVG handlers, XML stylesheets, external entities, and suspicious URLs must:

  • appear literally in <pre>.textContent;
  • create no corresponding DOM elements;
  • execute no script or dialog;
  • cause no object-content-originated request;
  • encounter no iframe, object, embed, HTML parser, or XML parser in Text Preview.

Permission and race tests

Verify:

  • no GetObject means no usable action and no retained body;
  • historical versions require their corresponding permission;
  • metadata and body use the same version ID;
  • a late old response cannot replace a new object’s preview;
  • 401, 403, 404, 416, and 5xx bodies never become preview content;
  • anonymous access and Console subpaths do not regress.

Browser regression

Use a real SILO/Console test instance to inspect both English and Chinese routes, light and dark themes, and narrow and desktop widths. Media, PDF, download, share, and version workflows require smoke coverage alongside the new text states.

Delivery and completion gates

The change belongs to pgsty/silo-console, even though the user report is tracked in the SILO server repository.

Delivery is staged:

  1. Merge the focused Console source and test change.
  2. Pass TypeScript checking, production build, automated matrices, and real-browser security regression.
  3. Update Console release notes and regenerate the actual embedded web assets.
  4. Publish a Console version; a minor release is appropriate for the new visible capability.
  5. Update SILO’s github.com/minio/console => github.com/pgsty/silo-console replacement to the exact new pseudo-version.
  6. Build a SILO candidate from that exact dependency and repeat integration checks.
  7. Publish the SILO binary and image, naming the first version that contains the feature.

These are separate states:

Gate Meaning
Console PR merged Implementation exists in source.
Console assets/tag published Console is independently consumable.
SILO dependency updated SILO main has integrated the change.
SILO release published Users can obtain the feature.

Issue #17 should not be described as fixed for users merely because a local preview or Console source PR exists.

Trade-off summary

The accepted design favors:

  • explicit scope over a generic browser viewer;
  • complete small files over partial large files;
  • source fidelity over automatic formatting;
  • strict UTF-8 over silent lossy decoding;
  • one inert text node over a full editor;
  • the existing download API over a new backend contract;
  • a verifiable security invariant over convenient same-origin rendering.

The cost is real: large logs and legacy encodings still require download, and the first release has no search, line numbers, wrapping toggle, or highlighting. Those omissions are deliberate. They make the feature small enough to audit and strong enough to trust.

Review record

The design was independently reviewed from three perspectives:

  • product scope, delivery, and acceptance;
  • security and frontend architecture;
  • compatibility and current-source verification.

The reviewers initially differed on MIME-only eligibility and lossy UTF-8 fallback. After cross-review they reached a single contract:

  • existing media classification wins;
  • text fallback accepts the four target extensions or four exact normalized MIME types;
  • HTML/XHTML extensions are explicitly excluded;
  • strict UTF-8 and NUL rejection are required;
  • lossy viewing is deferred to a separate proposal.

No unresolved design question remains. Implementation may proceed against this record.

Current Retry boundary: the too-large state permits reprobe of potentially stale listed size, never an unbounded download. The Range, response-header checks and at-most-1 MiB + 1-byte read limit still apply. Empty files are verified through the response path; unknown size must not be treated as zero.

4.16 - Go 1.27 TLS Defaults and OIDC Discovery Failure Modes

After SILO’s toolchain moved to Go 1.27, the Server TLS repair 48e184652 (“fix(tls): honor Go key exchange defaults across transports”) removed its explicit curve overrides. This page records the TLS changes and the diagnostic method for OIDC discovery failures that came out of issue #154, and the operational facts an administrator needs when identity goes missing at startup.

Release boundary, 2026-09-16: Server 20260903 already uses Go 1.27.1, but does not contain 48e184652. That later TLS repair is on main; upgrading the compiler and adopting this repair are separate changes.

Evidence class, stated up front. Every mechanism below is verified by synthetic experiments: ClientHello captures, fresh-process CA probes, and fixture reproductions. The #154 customer’s discovery URL and ingress configuration were never obtained, so no root cause is claimed for that deployment — two locally verified mechanisms could each produce the reported symptom, and either the ingress rejecting the new handshake, or a proxy rejecting the changed User-Agent, remains plausible. #154 stays open for an affected-environment retest.

What Go 1.27 changed

  • Explicit curve preferences now override the ML-KEM compat switches. GODEBUG=tlsmlkem=0 removes all ML-KEM hybrids from the default set; tlssecpmlkem=0 removes only the P-256/P-384 hybrids introduced in Go 1.26 and retains X25519MLKEM768. An application that configures CurvePreferences explicitly keeps ML-KEM in whatever list it names — a deliberate Go 1.27 change. SILO had eight TLS configuration points setting an explicit list including X25519MLKEM768; the fix removes all eight assignments and retires the helper, so these Server configuration points follow Go’s defaults and the compat switches work again. The stack review found pkg, mcli, and Console clients already used defaults; Console’s HTTPS listener retains its separate P-256-only policy.
  • ClientHello now offers ML-DSA signature algorithms (identifiers 0x09040x0906). ML-DSA is a signature scheme and distinct from ML-KEM: disabling hybrid key exchange does not disable the ML-DSA offer, and an ingress that rejects ML-DSA is not fixed by any ML-KEM switch.
  • The ClientHello grew. Measured on the same source and dependencies: Go 1.26.5 default 1497 bytes; Go 1.27.1 default 1509 bytes; the old explicit list under tlsmlkem=0 produced a 275-byte hello with no ML-KEM, while Go 1.27.1 with an explicit list still produced 1509 bytes containing ML-KEM. Rebuilding with a different compiler alone changed the handshake.
  • macOS root-CA behavior flips with the module’s Go directive. A fresh process honoring SSL_CERT_FILE/SSL_CERT_DIR instead of the Keychain is governed by the x509sslcertoverrideplatform GODEBUG default, which follows the main module’s go directive: go 1.26 modules ignore those variables on macOS (platform store wins), go 1.27 modules honor them — and the consuming application’s directive wins even when a library module is older. Operators on macOS should know that setting either variable replaces Keychain trust wholesale with the file/directory given; a stale or incomplete path then breaks chains the Keychain would have accepted, and unsetting restores the Keychain.
  • Not everything changed. TLS versions, cipher suites, certificate and hostname verification, proxy handling, and HTTP/2 selection are unaffected. The standard-library drain cap (256 KiB / 50 ms) and other audited Go 1.27 changes showed no SILO dependency. Go 1.27 binaries require macOS 13 or newer. Downgrading is not a supported path: the module graph requires Go ≥ 1.27.1 across Server, Console, and mc.

Why the OIDC-only patch was withdrawn

The investigation first produced a minimal candidate: clear CurvePreferences on the OIDC discovery transport only. It was deliberately not shipped. The same transport serves identity plugins, notification and lambda reachability checks, audit webhooks, and S3 cloud-backend tiers — fixing two OIDC call sites would have left every other consumer on the defective explicit list. The merged repair removes the explicit curves at all eight Server configuration points so the compat switches apply there, keeps certificate verification strict, and adds no protocol downgrade or automatic fallback. The archived one-transport patch must not be reapplied on top of the merged fix.

Diagnosing a discovery failure by phase

The startup chain is: server start → identity system init → fetch .well-known/openid-configuration (discovery) → fetch the jwks_uri keys → IAM store ready → Console initializes. Console’s own OIDC configuration dialog validates through the same server-side transport. A failure anywhere in the chain leaves IAM offline; a successful discovery does not clear the JWKS fetch, and a 503 on JWKS blocks IAM just as hard.

Discriminate by where the connection dies:

  • Reset right after the TLS ClientHello (tls_start then reset): suspect the ingress’s ClientHello handling — proxy CONNECT rules, TLS terminators, or anything keyed on hello size or contents. This is where the Go 1.27 changes land.
  • Reset after TLS completes (wrote_request then reset): the TLS layer is fine; look at HTTP-layer policy — WAF rules, User-Agent allowlists (the server’s UA changed from MinIO to Silo with the rebrand), routing. Replacing certificates or key exchange here has no targeted effect.
  • x509 errors: compare the chain actually received, the SNI, and the trust store the process resolves (see the macOS section above).
  • Always test from the same network position as the failing process — a fresh container does not inherit the failing container’s network namespace, and same-IP/same-proxy controls come first.

The health endpoint that tells the truth

/minio/health/live and /minio/health/ready both stay 200 while IAM is offline — readiness as deployed does not cover the identity system. The endpoint that reports it is /minio/health/cluster, which checks identity initialization and returns 503 with the X-Minio-Server-Status: iam-offline marker. Monitoring that should catch a broken IdP integration should probe the cluster health, plus one authenticated operation.

Recovery is automatic: identity initialization retries at randomized 0–3 s intervals, and a recovered IdP brings IAM back without a restart (observed sub-second to ~1.4 s locally). Retrying cannot fix a persistent incompatibility — a hello the ingress rejects stays rejected.

Transport facts worth knowing

The discovery/JWKS client builds its own transport: HTTP/2 disabled (no ALPN, HTTP/1.1), proxies taken only from HTTPS_PROXY/NO_PROXY (uppercase preferred; ALL_PROXY unused), DNS refresh defaulting to 30 s in Kubernetes/Docker and 10 min otherwise (overridable by the DNS cache TTL setting), dialing that walks addresses in order without shuffling, and timeouts of 5 s per TCP dial, 10 s for the TLS handshake, and 1 min to response headers. There is no total timeout on the discovery or JWKS fetch itself — a slow IdP can hold startup indefinitely; tightening that is a known, separately-sized follow-up.

Attribution

This record distills the issue #154 investigation and the September Go 1.27 stack review; the reproduction artifacts and the full evidence chain are retained outside the documentation tree. The supported statement is: the merged fix restores Go key-exchange defaults at the eight affected Server configuration points, verified with synthetic negative controls — it does not claim to have diagnosed any specific hidden deployment, and #154 remains open pending a retest in the affected environment.

Go behavior is grounded in the official 1.27 release notes and the actual toolchain. ClientHello byte counts above describe the recorded fixtures, not a fixed size for every connection.

4.17 - No I/O Before Auth, No Privilege From Headers

Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.

This record describes the CORS hot-path and replication-request trust repair merged into SILO as PR #101 (938603458 through 04b097fd9).

Status on 2026-09-03: PR #101 merged into main on 2026-09-01 with four follow-up commits: per-entry Snowball trust isolation (ff44527a3), request defaults preserved across Snowball workers (ab3ae99ca), and replication validity probes that verify the replication permissions (c9ad74673) under the rule prefix (5db7be4ee). Implementation, focused and race tests, the complete server package suite, object-lock tests, vet, build, two rounds of Fable 5 design review, repeated Opus 5 adversarial acceptance, and a real local TLS two-site replication run are complete. The pre-release cleanup kept the resident-only lookup with its fail-closed startup and load-failure states, dropped only the internal-namespace special case, and made the header-stripped request clone share the original request trailer so streaming-checksum uploads keep working for untrusted requests. Tag, package, image, deployment, and production verification remain separate gates.
Scope: HTTP request interpretation before and inside the S3 handlers. No S3 wire field, object format, bucket metadata format, replication protocol, encryption format, or client command changes.
Security properties: pre-authentication CORS processing performs no object-layer I/O; a header never grants replication semantics by itself; SSE-C ciphertext paths and replica-only metadata require both authentication and the corresponding replication permission.

Too Long; Didn’t Read (TL;DR)

Two bugs looked unrelated:

  1. an Origin header made the outermost CORS middleware treat the first URL segment as a bucket and synchronously load its metadata before authentication;
  2. X-Minio-Source-Replication-Request made downstream code believe a request was internal replication merely because the header existed.

They shared the same design failure: untrusted request shape was allowed to acquire expensive or privileged internal meaning before an authorization boundary.

The repair establishes two invariants:

before authentication:  parse cheaply; never load bucket metadata
after authentication:   derive one trust decision; downstream code consumes it

For CORS, the outer middleware now reads only metadata already resident in memory. For replication, handlers authenticate the original signed request first, authorize the appropriate replication action, and then attach a private trust decision to the request context. Untrusted internal headers are stripped only after signature verification. The context decision—not header removal—is the authority used by option builders, encryption paths, object lock, event generation, and metadata persistence.

Failure A: pre-authentication CORS amplification

corsHandler wraps the complete server router. Any request carrying Origin reaches it before S3 authentication, request validity checks, and the normal API limiter.

The per-bucket CORS implementation originally called the normal bucket metadata getter:

Origin-bearing request
  -> first URL segment becomes "bucket"
  -> GetCorsConfig
  -> GetConfig cache miss
  -> read .metadata.bin
  -> probe ten legacy config paths
  -> cache a default BucketMetadata record

When .metadata.bin did not exist, the loader intentionally searched legacy configuration files. With none found, it returned a valid empty metadata record rather than NoSuchBucket. The generic getter then inserted that record into metadataMap.

An unauthenticated client could therefore vary otherwise plausible names and obtain two effects per distinct value:

  • repeated erasure/object metadata reads before the normal request limiter;
  • growth of the in-memory bucket metadata map.

Name validation alone cannot repair this. An attacker can generate an effectively unbounded sequence of syntactically valid, nonexistent bucket names. Distributed deployments eventually prune stale map entries during the 15-minute metadata refresh; single-node deployments do not start that refresh loop, so their synthetic entries persist until restart.

Failure B: a marker header became authority

SILO and its MinIO-compatible clients use internal headers to preserve source state during replication. The most important marker is:

X-Minio-Source-Replication-Request: true

Before this repair, several paths treated header presence—or its raw string value—as proof that the request was a replication request. That affected more than metadata extraction:

  • GET of an SSE-C object could set NoDecryption and return ciphertext without the customer key to a caller holding only ordinary read permission;
  • source ETag and modification time could replace server-generated values;
  • source tagging, retention, and legal-hold timestamps could enter last-writer-wins comparisons;
  • a past object-lock retention date could be accepted through a raw marker check;
  • delete-marker identity and modification time could be supplied by the caller;
  • successful object events could be suppressed;
  • multipart actual size and encrypted checksum metadata could be injected at completion;
  • X-Amz-Replication-Status could be persisted from ordinary PUT, COPY, or POST-policy metadata extraction.

The earlier CVE-2026-34204 repair correctly stopped ordinary PUT and COPY from importing the replication SSE metadata that could make objects unreadable. It did not yet provide one authority shared by every reader of the marker, source fields, event state, object-lock exceptions, or multipart completion metadata.

Selected design

One exact marker, two trust levels

The marker is accepted only when it appears exactly once and its value is exactly lowercase true. Duplicate values, mixed case, and any other value are untrusted.

The handler then derives two related decisions:

Decision Requirements Semantics it may enable
trusted original request authenticated; non-anonymous principal; exact marker; s3:ReplicateObject or s3:ReplicateDelete on the addressed resource source ETag/MTime and source timestamps; actual size and encrypted checksum transfer; event and re-replication suppression; replication delete pool/version pinning
replicaTrusted trusted, plus raw request status REPLICA or a multipart upload whose stored status is REPLICA replica status persistence; replication SSE sealed-key import; SSE-C ciphertext/no-decryption path; replica-only object-lock behavior

The split is required by the real wire protocol. Not every legitimate replication request repeats X-Amz-Replication-Status: REPLICA.

The receiver follows this matrix:

Incoming shape Result
no marker ordinary S3 operation
marker without replication permission internal fields ignored; operation continues with ordinary semantics
REPLICA without replication permission 403 AccessDenied
exact marker + replication permission, no REPLICA trusted only
exact marker + replication permission + REPLICA trusted and replicaTrusted

The explicit 403 for an unauthorized REPLICA request prevents a claimed replica write from being silently downgraded into a new ordinary object that may be replicated again.

Authenticate the original, then sanitize

SigV4 signs request headers. Removing an internal header before authentication would change the canonical request and turn a valid signature into SignatureDoesNotMatch.

The ordering is therefore mandatory:

original request
  -> existing signature/authentication path
  -> ordinary S3 action authorization
  -> replication action authorization
  -> derive trusted / replicaTrusted
  -> bind decision to request context
  -> clone and strip untrusted internal fields
  -> option parsing, encryption, object lock, storage, events

The audit logger retains the original request. The effective request clone retains public S3, SSE, checksum, object-lock, copy-source, proxy, and replication-validity headers. It strips only internal source/replication controls, including source ETag/MTime/delete-marker/timestamps, replication SSE state, actual object size, encrypted checksum transfer, and the request use of X-Amz-Replication-Status.

Header stripping is defense in depth. All privileged consumers use the private context decision or an explicit Boolean; they do not infer trust by looking at the clone.

Replica status is not generic user metadata

X-Amz-Replication-Status is an S3 response header that MinIO-compatible servers also use as an internal request control. It no longer belongs to the generic supported-request-metadata list.

Ordinary PUT, COPY, multipart initiation, Snowball/PAX extraction, and POST policy cannot persist it merely by submitting the field. The receiver sets REPLICA explicitly only in a replicaTrusted branch.

This closes a subtle POST-policy path: a form field could previously store REPLICA, causing the resulting object to evade normal replication scheduling even though the POST principal never held replication permission.

Object lock receives an explicit decision

The object-lock parser used to accept past retention dates when the raw marker header was present. That package now receives allowPastRetainDate explicitly from replicaTrusted state.

The surrounding handler also uses the same decision when deciding whether an existing compliance/legal-hold version may be overwritten by a replica. This removes an internal-header dependency from the reusable object-lock package.

Actual replication wire matrix

The design was checked against the silo-go v7.3.1 emitter selected by the server’s go.mod, not inferred from comments or upstream documentation.

Operation Marker REPLICA on this request Receiver decision
regular replicated PutObject yes yes replicaTrusted
replicated NewMultipartUpload yes yes persist trusted multipart replica provenance
replicated PutObjectPart yes no trusted; replicaTrusted only when stored MPU status is REPLICA
replicated CompleteMultipartUpload yes no trusted; preserve source ETag/MTime, actual size, and encrypted checksum
CopyObject metadata replication yes yes replicaTrusted
replicated RemoveObject yes yes replicaTrusted with s3:ReplicateDelete
batch replication PUT/Complete yes no trusted; target credentials must hold s3:ReplicateObject
proxy/readiness/validity probes separate probe headers no marker authority probe behavior retained; those headers are never stripped by this repair

s3:ReplicateDelete is the trust gate, not the receiver’s only permission. For compatibility with deployed target policies, a trusted replication delete also requires s3:DeleteObject; an explicit deny on s3:DeleteObjectVersion still blocks a named-version purge. Ordinary clients do not use this compatibility path: an explicit UUID or versionId=null requires an allow for s3:DeleteObjectVersion.

Requiring REPLICA for every trusted operation would break PutPart, multipart completion, and batch replication. Trusting every marker would recreate the vulnerability. Stored multipart provenance bridges the two requirements for encrypted raw parts.

CORS resident-only state machine

The outer CORS middleware must remain cheaper than the request it is about to route. It now calls a dedicated resident-only getter that takes one read lock and examines only in-memory state.

Bucket metadata state CORS result Object-layer work
resident, valid per-bucket CORS apply per-bucket rule; a failed refresh keeps the last loaded document, as for every other bucket configuration none
resident, no CORS document use global CORS fallback none
resident, invalid stored CORS fail closed; continue without CORS headers and log once none
not resident while startup loading is still running fail closed none
not resident after startup: real bucket whose metadata failed to load fail closed none
not resident after startup: reserved, invalid, internal, or unknown name global fallback none

The lookup consults the resident map and a bounded set of real buckets whose metadata failed to load at startup or during a refresh. That set is filled only from disk-derived bucket lists, never from a client path, never records a bucket that is already resident, and a successful load, Set, bucket removal, stale-bucket reconciliation, and subsystem reset clear it. Both non-resident states fail closed: a presigned URL is authenticated by its own signature, so the bucket’s CORS document is the only origin boundary a browser enforces for it, and answering with the global policy would let a leaked URL be used from any origin. The internal .minio.sys namespace no longer has a special case; like any reserved or invalid name it is not a bucket, gets the global fallback, and is rejected downstream.

Alternatives rejected

Alternative Why it was rejected
Validate bucket names before the old CORS getter valid nonexistent names still provide an unbounded attacker-controlled key space and still trigger pre-auth I/O
Call GetBucketInfo before loading CORS replaces eleven metadata reads with at least one unthrottled backend operation per attacker name
Cache every negative result with a TTL bounds duration, not attacker cardinality or the initial I/O amplification
Strip replication headers before authentication breaks SigV4 canonical-request verification
Reject every request carrying an internal marker turns formerly ignored extra headers into broad client failures and breaks legitimate marker-only replication calls
Require REPLICA on every trusted call breaks replicated PutPart, CompleteMultipartUpload, and batch replication wire behavior
Let every handler re-check raw headers independently recreates inconsistent trust rules and leaves future consumers easy to miss
Store a Boolean in ObjectOptions but leave events/object lock on headers produces two authorities that can disagree; the original bug class remains

Implementation boundary

The selected change is intentionally layered:

  1. a small request-trust module defines exact marker parsing, replication authorization, private context state, and the post-authentication effective request;
  2. object option builders parse source fields only when their caller provides trusted state;
  3. DecryptObjectInfo, event request parameters, multipart completion, delete options, and object lock consume the same decision;
  4. handlers calculate trust immediately after their existing authentication path;
  5. multipart part handling combines current-request trust with stored MPU replica provenance;
  6. generic metadata extraction does not accept replica status;
  7. CORS middleware uses a separate resident-only metadata accessor and never calls the load-on-miss getter.

No object-layer API needs to infer HTTP trust. Programmatic internal callers that construct ObjectOptions{ReplicationRequest: true} remain unchanged.

Verification and adversarial review

Regression coverage includes:

  • hundreds of distinct valid missing bucket names, both actual and preflight CORS requests, with zero metadata reads and no map growth;
  • Console, reserved, invalid, startup, internal namespace, and invalid stored CORS paths;
  • least-privilege SSE-C GET, HEAD, and GetObjectAttributes callers with correct, missing, wrong-case, and unauthorized markers;
  • marker-only batch-style PUT preserving source ETag/MTime only with s3:ReplicateObject;
  • unauthorized REPLICA PUT and DELETE returning 403;
  • POST policy unable to forge replica status;
  • object-lock past-date parsing with and without replica trust;
  • marker-only CopyObject with SSE-C source headers copying plaintext rather than ciphertext;
  • fake marker on an ordinary SSE-C MPU failing instead of storing raw bytes;
  • a real in-process SSE-C multipart replication chain: encrypted source, raw ciphertext part, trusted replica initiation, marker-only PutPart and Complete, and exact plaintext recovery with the original key.

The final local tree passed focused and race tests, the complete cmd suite, object-lock tests, vet, build, and diff checks.

A separate black-box run started two TLS-enabled SILO instances built from the candidate and enabled real site replication. It verified:

  • an SSE-C 4 KiB object;
  • an SSE-C 12 MiB, three-part multipart object;
  • an SSE-C CopyObject result;
  • a replicated delete marker.

Source and target ETag, size, version ID, SSE-C key MD5, decrypted SHA-256, and delete-marker version ID matched; targets reported REPLICA.

Two Fable 5 review rounds first corrected the trust model for marker-only batch and multipart calls, then audited the implementation. A final independent Claude Code Opus 5 review reported GO, with no P0/P1 findings, and independently reran build, vet, race, object-lock, and full cmd tests.

Compatibility and operations

  • Ordinary clients: no request change. Untrusted internal headers are ignored instead of acquiring internal semantics.
  • Unauthorized claimed replica writes: requests carrying X-Amz-Replication-Status: REPLICA now return 403 where some multipart subpaths previously lacked a uniform check.
  • Batch replication: destination credentials must include s3:ReplicateObject, as documented in the batch replication requirements. Without it, the receiver processes marker-only writes as ordinary writes and does not preserve source ETag/MTime.
  • SSE-C: ordinary reads still require the customer key. Authorized replica reads may use the raw ciphertext path needed to preserve encrypted bytes.
  • Events: only trusted replication suppresses replica creation/access events; a forged marker no longer silences them.
  • Object lock: replica exceptions are permission-derived rather than header-derived.
  • Performance: CORS removes pre-authentication backend work. Trusted writes add policy checks already required by the replication contract; no additional object pass is introduced.
  • Rolling upgrade: wire and storage formats are unchanged. New receivers enforce the trust boundary; old receivers remain vulnerable to the old header semantics until upgraded. Per-bucket CORS behavior can therefore differ by node during the rolling window.
  • Rollback: data written by the repaired version remains readable by the previous version, but rollback reopens both trust defects and restores pre-authentication metadata loads.

Residual risks and follow-ups

  • 2026-09-09 replication reliability follow-up: Delete completion, MRF visibility, and resync cancellation records the reproductions, minimal fixes, Fable review, and PR #162 validation for #153, #152, and #137. It addresses reliability after trusted requests enter the replication pipeline, preserving this page’s authorization boundary.

  • Emit a rate-limited diagnostic when a marker-bearing request lacks replication permission; the safe ordinary fallback is otherwise easy to misdiagnose as an ETag/MTime mismatch.

  • Replication validity probes now verify the replication permissions the target credentials need and place the synthetic validation key under the rule prefix (c9ad74673, 5db7be4ee).

  • This review covers the named source/replication headers. Other future internal controls must still answer the same question: which authenticated decision allowed this client value to acquire internal meaning?

Conclusion

An internal-looking header is still client input. A bucket-shaped URL segment is still attacker input. The durable repair is to stop either one from becoming authority by accident:

Before authentication, do no backend work. After authentication, derive trust once and pass the decision—not the claim—downstream.

That rule is broader than CORS or replication. It is the boundary future SILO handlers should preserve whenever inexpensive public request syntax meets expensive or privileged internal state.

This repair is SN-2026-008 in the ledger, with delivery in the 20260903 chronicle. The silo-go reference above describes the original temporary dependency. After 0079723d3, Server returned to verified upstream minio-go; the component matrix records selected versions.

4.18 - One Endpoint, Two Privileges: Separating User and Group Status

Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.

This document records the discussion, repair, and final authorization design for upstream issue minio/minio#21478 and SILO PR #73.

Status on 2026-08-26: SILO PR #73 was merged as 2e2377d1c, preserving the signed-off repair commit 58735ee38. All eight reported checks passed. Upstream issue #21478 and PR #21482 remain open, but minio/minio is archived and read-only, so no further issue comment or merge can be made there.
Group follow-up on 2026-08-28: final release review found the same fixed-action defect in set-group-status. Signed-off server commit 229fe2b3c now selects admin:EnableGroup or admin:DisableGroup from the requested target state and adds a real four-way IAM authorization test. Local verification and independent review are complete; it was merged into main on 2026-08-29, and tag and delivery remain pending.
Scope: authorize enabling and disabling a user with their respective existing Admin Actions. Do not change the route, status values, account storage, replication record, or client API.
Security property: possessing admin:DisableUser must not grant the ability to enable an account, and possessing admin:EnableUser must not grant the ability to disable one.
Release boundary: merge, tag, release package, container image, deployment, and production verification remain separate gates.

Too Long; Didn’t Read (TL;DR)

SILO exposes both admin:EnableUser and admin:DisableUser, but the shared set-user-status handler historically authorized every request with admin:EnableUser. A policy that granted only admin:DisableUser therefore could not disable an account. The workaround was to grant admin:EnableUser as well, which destroyed the least-privilege boundary that the two action names promised.

The selected repair derives exactly one required action from the requested target state before authorization:

Requested status Required action
enabled admin:EnableUser
disabled admin:DisableUser
invalid or unknown admin:EnableUser, preserving the previous authorization-before-validation default

The handler then calls validateAdminReq once. A four-way IAM test proves both positive operations and both denied cross-action operations. This is intentionally stricter than preserving the accidental historical behavior in which an Enable-only policy could also disable users.

The same rule now applies to group status:

Requested group status Required action
enabled admin:EnableGroup
disabled admin:DisableGroup
invalid or unknown admin:EnableGroup, preserving the previous authorization-before-validation default

Before the follow-up, an EnableGroup-only principal could disable a group, while a DisableGroup-only principal received AccessDenied for that exact operation. The group repair uses the same one-selector, one-authorization design rather than treating the two actions as aliases.

The reported defect

The Admin API uses one route for both state transitions:

PUT /minio/admin/v3/set-user-status
    ?accessKey=<target>
    &status=enabled|disabled

Before the repair, the handler checked one fixed action before reading the requested status:

objectAPI, creds := validateAdminReq(ctx, w, r, policy.EnableUserAdminAction)

The later call to SetUserStatus correctly received either enabled or disabled, but authorization had already treated both as Enable operations. admin:DisableUser existed in the policy vocabulary and documentation while being ineffective for this endpoint on its own.

Issue #21478 supplied the practical counterexample: an operator wanted a policy that could disable accounts during an incident without being able to restore them. A policy containing admin:DisableUser received AccessDenied; adding admin:EnableUser made the request work, but also gave the operator the more powerful recovery transition that the policy intentionally withheld.

This is not a missing convenience permission. It is a mismatch between the policy model and the enforcement point:

policy says:     DisableUser only
request says:    target state = disabled
handler checked: EnableUser
result:          legitimate disable denied
workaround:      grant an unwanted enable capability

Why two actions must mean two capabilities

An account state transition has direction. Disabling is commonly delegated to incident responders, fraud controls, compliance automation, or a break-glass process. Enabling restores access and may require a separate approver.

If either action authorizes both transitions, a policy author cannot express that separation. The server would publish two names while enforcing one combined capability. The design contract is therefore strict:

Principal policy Disable target Enable target
admin:DisableUser only allow deny
admin:EnableUser only deny allow
both actions allow allow
neither action deny deny

The built-in consoleAdmin policy grants admin:*, so full administrators retain both operations. The compatibility impact is limited to custom restricted policies that relied on the old accidental behavior.

The public PBAC reference now states the same contract for admin:EnableUser and admin:DisableUser.

Design goals and non-goals

Goals

  1. Make both existing Admin Actions enforceable according to their names.
  2. Preserve least privilege in both directions.
  3. Perform one authorization decision and write at most one authorization error.
  4. Preserve the route, request values, response format, self-mutation guard, IAM storage call, and site-replication hook.
  5. Encode the contract in tests that fail if the two permissions are broadened or swapped again.

Non-goals

  • split the endpoint into separate enable and disable routes;
  • add a new combined action or change policy syntax;
  • change user status persistence or replication;
  • redesign Console permissions;
  • infer release, image, deployment, or production delivery from a source merge.

Alternatives considered

Keep checking admin:EnableUser for both states

This preserves behavior but leaves admin:DisableUser unusable and forces over-privileged policies. It is the defect, not a compatibility contract worth retaining.

Require both actions for either transition

This makes the two labels decorative and prevents delegated disable-only operation. It is stricter in quantity but weaker in expressiveness and least privilege.

Try Enable authorization, then retry Disable authorization

Upstream PR #21482 attempted this shape for a disabled request. It first called validateAdminReq with EnableUser, then called it again with DisableUser if the first result was nil.

That helper has an important contract: when it returns a nil object layer, it has already written an error response. A Disable-only request can therefore commit a 403 response before the second authorization succeeds and the handler proceeds to mutate account state. Authorization fallback must never continue after an error response has been committed.

Accept either Enable or Disable for a disabled request

validateAdminReq already accepts multiple actions and succeeds if any one is allowed, so compatibility behavior could be implemented safely with one variadic call. That would let Disable-only policies work while preserving the historical ability of Enable-only policies to disable.

SILO rejected this option because the historical ability was the enforcement bug. It would solve the reporter’s positive case but retain a cross-action privilege that contradicts the two-action model. Operators who want both transitions can grant both actions explicitly.

Validate the status before authenticating

Rejecting unknown status values first would change error precedence: a caller that previously had to pass the Enable authorization gate could now receive a validation result before authorization. The repair does not need that broader behavioral change.

Unknown values therefore retain admin:EnableUser as the authorization default. Valid disabled is the only value that selects admin:DisableUser; the existing IAM layer remains responsible for rejecting invalid status values after authorization.

The selected implementation

The repair adds a pure selector:

func setUserStatusAdminAction(status string) policy.AdminAction {
    if madmin.AccountStatus(status) == madmin.AccountDisabled {
        return policy.DisableUserAdminAction
    }
    return policy.EnableUserAdminAction
}

The handler reads the route variables, selects the action, and authorizes exactly once:

vars := mux.Vars(r)
accessKey := vars["accessKey"]
status := vars["status"]

objectAPI, creds := validateAdminReq(ctx, w, r, setUserStatusAdminAction(status))
if objectAPI == nil {
    return
}

Everything after the gate remains unchanged:

  • a caller still cannot enable or disable its own account;
  • globalIAMSys.SetUserStatus validates and persists the requested status;
  • site replication records the same status and timestamp;
  • response and audit behavior use the existing path.

The selector depends only on the requested target state. It does not load the current user, infer a transition from stored state, or make authorization depend on whether the target exists. This keeps authorization deterministic and avoids a read-before-authentication dependency.

Why the repair is safe

The correctness argument consists of five invariants:

  1. Every valid status maps to exactly one Admin Action.
  2. validateAdminReq is invoked once, so a failed authorization cannot be followed by mutation.
  3. The mutation call is reachable only after the selected action succeeds.
  4. Invalid status values preserve the old Enable authorization boundary and are still rejected by the existing status-validation path.
  5. No storage, replication, wire, or client contract changes; only the permission required to reach the existing mutation changes.

The change is a deliberate authorization tightening for Enable-only custom policies that used the disable operation. That tightening is the mechanism that makes admin:DisableUser a real independent capability.

Test design

Pure action mapping

The unit test fixes three selector cases:

Input Expected action
enabled EnableUser
disabled DisableUser
invalid legacy EnableUser default

Four-way IAM authorization matrix

The integration test creates separate users and policies, then exercises the real Admin API:

  1. a Disable-only client successfully disables a target;
  2. the same client receives AccessDenied when enabling it;
  3. an Enable-only client successfully enables the target;
  4. the same client receives AccessDenied when disabling it.

Positive assertions alone would not prove least privilege: both policies could accidentally authorize both states and still pass. The two negative cross-action assertions are the security regression tests.

The test removes every temporary user and policy after execution. It runs inside the existing IAM server suite, so it covers request signing, policy attachment, handler authorization, persistence, and Admin-client error decoding rather than testing only the helper.

Repair and verification record

The server checkout originally contained unrelated dependency, generated-credit, checksum-test, and security-document changes, while local main was behind the remote. The two user-status files were isolated into a clean worktree based on current origin/main; no unrelated file entered the repair commit.

Local verification passed:

go test ./cmd -run '^TestSetUserStatusAdminAction$' -count=1
go test ./cmd -run '^TestIAMInternalIDPServerSuite$' -count=1
git diff --check

The signed-off commit 58735ee38 was pushed in PR #73. Its eight remote checks all passed:

  • DCO sign-off;
  • format, build, and vet;
  • lint and generated files;
  • cmd/ tests;
  • internal/ tests;
  • race detector and S3 Select;
  • cross compilation;
  • vulnerability analysis.

The PR was merged with the repository’s normal merge strategy as 2e2377d1c. Local main was then fast-forwarded only after the two original working files were byte-for-byte and patch-ID identical to the merged result. The unrelated local changes remained intact, and the temporary worktree and task branch were removed after the code became recoverable from main and PR #73.

Least-privilege policy examples

Disable-only operator

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "admin:DisableUser",
        "admin:GetUser"
      ]
    }
  ]
}

This principal can inspect and disable another user, but cannot enable it.

Enable-only operator

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "admin:EnableUser",
        "admin:GetUser"
      ]
    }
  ]
}

This principal can inspect and enable another user, but cannot disable it. Grant both actions explicitly to roles responsible for the complete account lifecycle.

Group-status follow-up

The group endpoint has the same shape as the user endpoint:

PUT /minio/admin/v3/set-group-status
    ?group=<target>
    &status=enabled|disabled

It also publishes two existing actions, admin:EnableGroup and admin:DisableGroup. The inherited handler nevertheless authorized every request with EnableGroup before reading status. This was not merely a dead permission: it reversed least privilege in both directions. The wrong principal could disable a group, and the intended disable-only principal could not.

The follow-up adds setGroupStatusAdminAction, deliberately matching setUserStatusAdminAction:

func setGroupStatusAdminAction(status string) policy.AdminAction {
    if madmin.GroupStatus(status) == madmin.GroupDisabled {
        return policy.DisableGroupAdminAction
    }
    return policy.EnableGroupAdminAction
}

The integration test creates separate EnableGroup-only and DisableGroup-only administrators and a real target group. It proves:

  1. DisableGroup-only can disable;
  2. DisableGroup-only cannot enable;
  3. EnableGroup-only can enable;
  4. EnableGroup-only cannot disable.

The suite exercises signed Admin requests, policy attachment, handler authorization, IAM mutation, response decoding, and cleanup. Invalid status still selects the legacy Enable action before the existing validation error, so the change does not expose a new pre-authentication oracle. The successful site-replication hook remains after mutation and is not called for denied requests.

This follow-up changes no user behavior and introduces no new policy action. It makes the two already documented group actions enforce the same state-specific contract as their user counterparts.

Compatibility and migration

No client or API migration is required. The endpoint, query parameters, status strings, success response, and Admin-client method are unchanged.

Policy review is required for restricted administrative roles:

  • a role that should only disable users needs admin:DisableUser;
  • a role that should only enable users needs admin:EnableUser;
  • a role that must do both needs both actions;
  • consoleAdmin and other admin:* policies are unaffected;
  • a legacy custom policy containing only admin:EnableUser can no longer use that permission to disable users and must add admin:DisableUser if both operations are intended.

The equivalent rules now apply to group-management roles:

  • a role that should only disable groups needs admin:DisableGroup;
  • a role that should only enable groups needs admin:EnableGroup;
  • a role that must do both needs both actions;
  • a legacy EnableGroup-only role can no longer disable groups.

This is a source-level compatibility change in authorization behavior, not a wire-protocol break.

Upstream disposition

As of this record, upstream issue #21478 and PR #21482 are still displayed as open. The upstream repository is archived and read-only. An attempt to leave the single-authorization analysis on the PR was rejected by GitHub because archived, locked discussions cannot accept comments.

The upstream artifacts remain useful provenance but are no longer an actionable delivery path. SILO owns its implemented semantics, tests, merge, release note, and eventual production verification.

Delivery state

Gate User repair Group follow-up on 2026-08-28
Design decision complete complete
Implementation and local tests complete complete
Independent adversarial review complete complete, GO
Signed-off commit complete 229fe2b3c on main
Push, PR CI, and merge complete merged 2026-08-29
Tagged SILO release Server 20260903 Server 20260903
Release package or container image See the 20260903 release record See the 20260903 release record
Deployment not established not established
Production behavior not established not established
Upstream merge unavailable; repository archived not applicable

Conclusion

The repairs make the authorization model tell the truth. Enabling and disabling users or groups are opposite state transitions with different operational risk, and SILO already exposes different policy actions for each direction. Each handler must therefore select the action from the requested target state and authorize once before mutation.

The code change is small because the design boundary is clear. The durable result is larger: an explicit permission matrix, rejected compatibility alternatives, an invalid-input rule, a four-way integration test, a clean merge record, migration guidance, and an honest release boundary.

The enable/disable repair is separate from the later admin:ChangeMyPassword split. Before upgrading the September candidate, preserve the paired Deny needed for an existing self-password restriction using the password migration guide.

4.19 - Config Environment Files Are Not Shell Scripts

Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.

This record defines the startup contract for MINIO_CONFIG_ENV_FILE and explains the compatibility repair committed in SILO as 2aea7fe9c.

Status on 2026-08-28: implementation, focused tests, the complete cmd and internal suites, tagged tests, race tests, vet, lint, generated-file checks, rebrand guards, build, and an independent local Fable Max review are complete. The commit was merged into main on 2026-08-29 as 2aea7fe9c; tag, package, image, deployment, and production verification remain separate gates.
Scope: environment-file parsing and named-target discovery only. No configuration key, subsystem, value, precedence, storage format, or client API changes.
Compatibility rule: the file is a SILO input format. Supporting an optional export prefix does not make it a POSIX shell program.

Too Long; Didn’t Read (TL;DR)

SILO can load startup variables from a file:

export MINIO_CONFIG_ENV_FILE=/etc/default/silo
silo server /data

The parser accepts assignments such as:

MINIO_ROOT_USER = silo-admin
MINIO_ROOT_PASSWORD = "  significant surrounding spaces  "
MINIO_NOTIFY_WEBHOOK_ENABLE_my-hook = off
MINIO_NOTIFY_WEBHOOK_ENDPOINT_my-hook = https://events.example.com/minio

The last two names are important. Multi-target configuration appends the target name verbatim after an underscore. The configuration subsystem does not restrict a target to a shell identifier; names containing -, ., :, digits, or printable Unicode can be discovered and resolved exactly.

A hardening change accidentally validated every key as [A-Za-z_][A-Za-z0-9_]*. It made my-hook invalid and stopped the server during restart even though the previous loader and the configuration target model accepted it. The repair validates what SILO actually needs instead:

  • the name is non-empty, valid UTF-8, and made of visible non-whitespace characters;
  • = and NUL are not allowed in a name;
  • NUL is not allowed in a value;
  • invalid input reports file and line without reporting the value;
  • the complete file is parsed before any assignment is applied.

Why the regression was real

The environment-file loader calls os.Setenv after parsing. An operating-system environment is a list of strings, not a shell variable namespace. Shell assignment syntax is narrower because the shell must tokenize and expand variable names in its own language.

Named SILO configuration targets are built differently:

MINIO_<SUBSYSTEM>_<PARAMETER>_<target>

For example:

MINIO_NOTIFY_WEBHOOK_ENABLE_my-hook
MINIO_NOTIFY_WEBHOOK_ENDPOINT_my-hook

Target discovery lists variables by the fixed parameter prefix and treats the remaining suffix as the target name. Target lookup reconstructs the same name without uppercasing or sanitizing that suffix. Rejecting - in the file parser therefore broke a valid discover-to-resolve path; it did not protect a shell evaluation path because no shell evaluates the file.

The failure is operationally sharp. MINIO_CONFIG_ENV_FILE is loaded only at startup. A server can continue running with an old process environment, then fail on its next restart after the file or binary changes. Startup must fail on malformed input, but it must not invent a narrower target grammar than the configuration system.

The file grammar

Lines and comments

  • blank lines are ignored;
  • a line whose first non-whitespace character is # is ignored;
  • an optional standalone export followed by whitespace is removed;
  • exportFOO=value remains the key exportFOO; it is not mistaken for the prefix;
  • the first = separates key and value, so additional = characters remain part of the value.

The file is not a shell. It does not perform variable expansion, command substitution, backslash processing, or inline-comment interpretation.

Keys

Surrounding whitespace around the key is removed. The remaining key must:

  1. be non-empty valid UTF-8;
  2. contain only Unicode graphic characters;
  3. contain no whitespace, =, NUL, control, or invisible format characters.

This preserves OS-compatible names and multi-target suffixes while rejecting visually empty or structurally ambiguous keys. A key beginning with a digit or punctuation is accepted by the parser; SILO still reads only the exact names used by its configuration and runtime components.

Values and quoting

Unquoted values are trimmed. To retain leading or trailing spaces, quote the complete value with matching single or double quotes:

PLAIN = value
SPACED = "  value with significant spaces  "
TOKEN = scheme://user:[email protected]?a=b
EMPTY =

The parser removes one matching outer quote pair. It does not interpret escapes inside the quoted value. NUL is always rejected because it cannot be represented in an environment entry.

Failure and secrecy contract

Syntax errors stop startup. Diagnostics include the file path, line number, and the invalid key or error class, but never the value. A password on a malformed line must not be copied into logs.

Parsing is all-or-nothing: a syntax error returns no entries, and assignment starts only after the complete file has parsed. If the operating system rejects a validated assignment, SILO also stops startup and identifies the key and file. Since the process exits, it never serves requests with a partially loaded environment.

The file itself remains a privileged secret-bearing input. Operators must protect it with appropriate ownership and mode; parser validation is not a substitute for filesystem permissions.

Regression matrix

The committed tests cover:

  • spaces and tabs around =;
  • quoted values with significant spaces;
  • standalone export, including Unicode whitespace after it;
  • keys beginning with _, a digit, or punctuation;
  • named targets using -, ., :, and Unicode;
  • exact named-target discovery through the configuration subsystem;
  • empty keys, whitespace, NUL, and invisible format characters;
  • NUL values;
  • multiple = characters in URLs and tokens;
  • file-and-line diagnostics that redact values;
  • all-or-nothing parse results.

The implementation passed the complete local server verification matrix and a read-only adversarial review. Windows-specific os.Setenv behavior has not been exercised on a Windows runner; unsupported platform rejection remains fail-fast rather than silent.

Compatibility and delivery

No configuration migration is required. Existing ordinary environment names behave unchanged. Files using shell-style whitespace become more predictable, and previously accepted named targets work again.

The visible compatibility changes are intentional:

  • invalid or invisible names now fail instead of being silently ignored;
  • unquoted surrounding value whitespace is trimmed; quote it when significant;
  • malformed input stops startup with a redacted location-aware error;
  • a valid punctuation-bearing target is no longer rejected merely because a shell could not assign it with NAME=value syntax.

The parser contract is included in Server 20260903. Verify the artifact running in each deployment separately.

Conclusion

Configuration compatibility depends on validating the format SILO actually consumes. MINIO_CONFIG_ENV_FILE borrows a small amount of dotenv-like syntax for operator convenience, but it is not executed by a shell. The repair restores named-target compatibility while retaining strict NUL, invisibility, redaction, and fail-fast guarantees.

The original #65 also exposed a restart trap: the old parser stored KEY = new as a trailing-space key KEY . A systemd cold start could work because systemd parses EnvironmentFile itself; an admin API re-exec inherited KEY=old, and the malformed new key did not replace it. Restart could succeed with stale credentials or KMS/IdP settings. The repair makes whitespace assignments take effect, so review their intended values before upgrading.

4.20 - Two SSE-C Keys, One CopyObject Response

Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.

This record explains the CopyObject SSE-C checksum response repair committed in SILO as e73436c99.

Status on 2026-08-28: implementation, encryption and key-rotation tests, complete server suites, race tests, static checks, build, and independent Fable Max acceptance review are complete. The commit was merged into main on 2026-08-29 as e73436c99; tag, package, image, deployment, and production verification remain separate gates.
Scope: the successful CopyObject XML and HTTP response after the destination object has committed. Stored object bytes, checksum metadata, encryption format, source decryption, federation, replication, and historical objects are unchanged.
Security property: source SSE-C headers may decrypt only source state; destination SSE-C headers may decrypt only committed destination state.

Too Long; Didn’t Read (TL;DR)

An SSE-C copy can use two independent keys:

Role Request headers Purpose
source X-Amz-Copy-Source-Server-Side-Encryption-Customer-* decrypt the source object
destination X-Amz-Server-Side-Encryption-Customer-* encrypt and later interpret the committed destination object

SILO correctly wrote the destination with its destination key. However, after commit, both the XML generator and the generic PUT-response header helper received the complete CopyObject request. The checksum metadata decrypter intentionally prefers copy-source SSE-C headers when they are present. That priority is correct while reading the source, but wrong when interpreting the committed destination.

With source key A and destination key B:

source body       --decrypt A--> logical bytes
logical bytes     --encrypt B--> committed destination
destination csum  --seal B-----> stored checksum metadata
response decoder  --try A------> key mismatch, checksum omitted

The object and stored checksum were correct; only the successful response was incomplete. The repair constructs a destination response-header view by removing exactly the three copy-source SSE-C customer headers. It decrypts the destination checksum once, then reuses the resulting map for both XML and HTTP response headers.

Observable failure

The failure requires a checksum-bearing destination and distinct source/destination SSE-C contexts. A representative request supplies:

x-amz-copy-source: /bucket/source
x-amz-copy-source-server-side-encryption-customer-algorithm: AES256
x-amz-copy-source-server-side-encryption-customer-key: <key A>
x-amz-copy-source-server-side-encryption-customer-key-md5: <md5 A>
x-amz-server-side-encryption-customer-algorithm: AES256
x-amz-server-side-encryption-customer-key: <key B>
x-amz-server-side-encryption-customer-key-md5: <md5 B>
x-amz-checksum-algorithm: CRC32

Before the repair:

  • CopyObject returned HTTP 200;
  • reading the destination with key B returned the correct body;
  • stored destination checksum metadata decrypted with key B and matched the logical bytes;
  • the CopyObject XML and HTTP response omitted CRC32 and ChecksumType.

This is a response-contract defect, not evidence of corrupted object data.

The same ambiguity affects same-object SSE-C key rotation. After metadata has been resealed under key B, the request still carries source key A in the copy-source headers. Response generation must describe the post-rotation object, so it must use B.

Why the global decrypter must not change

The metadata decrypter’s copy-source priority is not itself a bug. Earlier in CopyObject, the server examines source checksum metadata to decide whether to preserve its algorithm, recompute a full-object value, or add the default CRC64NVME checksum. For an SSE-C source, that metadata is protected by the source object key and therefore requires the copy-source headers.

Changing the global priority to prefer destination SSE-C headers would fix the final response while breaking source checksum interpretation. The safe boundary is temporal and object-specific:

before commit: full request headers, source context
after commit:  destination-only SSE-C headers, destination context

The repair applies only at that post-commit boundary.

Selected implementation

Destination response view

The handler clones the request headers and removes exactly:

  • X-Amz-Copy-Source-Server-Side-Encryption-Customer-Algorithm;
  • X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key;
  • X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key-MD5.

Regular destination SSE-C headers remain. SSE-S3 and SSE-KMS destination metadata needs no customer key and continues through the existing path.

Decrypt once, project twice

Before the repair, CopyObject called decryptChecksums once while building XML and again while writing success headers. For SSE-S3 or SSE-KMS this could repeat KMS unseal work.

The repaired flow is:

committed ObjectInfo
  -> decryptChecksums(destination headers) once
  -> CopyObjectResult XML fields
  -> x-amz-checksum-* and x-amz-checksum-type response headers

The generic setPutObjHeaders wrapper remains available to PutObject, CompleteMultipartUpload, and DeleteObject. CopyObject calls a narrow helper that accepts the already decrypted checksum map. ETag, VersionID, delete-marker, lifecycle prediction, and checksum header behavior remain in one shared implementation.

Regression matrix

The tests cover:

  • plaintext source to SSE-C destination;
  • compressed and uncompressed SSE-C destinations;
  • SSE-C source key A to destination key B;
  • checksum value and type in both CopyObject XML and HTTP headers;
  • stored checksum decrypted with destination key B;
  • destination body readable with B;
  • same-object key rotation from A to B;
  • checksum response after rotation;
  • SSE-S3 source and destination combinations;
  • all object-layer backends used by the API test harness.

The final combined tree passed focused encryption tests, the complete cmd and internal suites, the project’s tagged test configuration, full go test -race ./..., vet, lint, generated-file checks, rebrand guards, and a local build. A mirror Fable Max review reported no P0–P2 findings and independently confirmed that source decryption still receives the full request while destination response decryption receives the filtered view.

Compatibility and operational impact

  • Successful CopyObject responses: checksum fields that were previously missing now appear when the committed destination has a checksum.
  • Stored objects: no rewrite, migration, metadata-format, or encryption-format change.
  • Existing objects: unaffected; the defect existed only in the one-time successful response.
  • Clients: no request change. Clients already providing both source and destination SSE-C keys receive a more complete S3-compatible result.
  • Performance: one metadata checksum decryption instead of two; no additional object read or hash pass.
  • Rolling upgrade: old nodes may omit the fields while new nodes return them. Stored objects remain mutually readable.
  • Rollback: restores response omission but does not damage objects created while the repair was present.
  • Security: no key or digest value is added to logs or error responses. The response carries only the checksum already authorized for the successful write.

This repair does not resolve the separately deferred legacy federation CopyObject branch and does not audit or modify historical compressed-object checksums. Those questions have different data and operational boundaries.

Conclusion

CopyObject is one request with two object identities. Reusing the full request after commit erased that distinction: a source key was allowed to shadow the destination key while describing destination metadata. The durable repair is not a new encryption scheme; it is an explicit context boundary, followed by one decryption and two faithful response projections.

4.21 - Why CompleteMultipartUpload Must Return ChecksumType: Review of PR #57

PR #57, contributed by Shooks (@Dansyuqri), fixed #47. The repair is included in Server 20260903. This records the response contract; production deployment remains specific to the artifact and installation an operator actually runs.

The defect and its scope

Investigation of @cbornet’s #31 separated two defects. Multipart CRC32 completion itself was repaired by c8590413f and 3e14733f1. After successful completion, the object retained its checksum type and HEAD could return it, but completion XML omitted that type. SDK callers therefore saw a missing ChecksumType beside a valid checksum value.

This omission did not corrupt stored data. It made a full-object checksum and a composite checksum harder to distinguish. Their Base64 encodings do not tell a consumer which calculation to reproduce.

Response contract

Stored state Completion response
Full-object checksum ChecksumType=FULL_OBJECT with its algorithm value
Composite multipart checksum ChecksumType=COMPOSITE with its algorithm value
No additional checksum No ChecksumType element; no invented checksum

The S3 completion API defines the two type values. ETag is a separate field and is not a substitute for the additional S3 checksum.

Implementation and evidence

The production change adds ChecksumType string with xml:"ChecksumType,omitempty" to CompleteMultipartUploadResponse and assigns cs[xhttp.AmzChecksumType] after decoding the stored checksum metadata. The generator reuses the same state as the other checksum APIs; it does not recalculate content or infer a type from part count.

The merged PR contains response tests for full-object, composite and absent checksums. The original change also registered the exported field in the then-current rebrand inventory. That inventory’s exported-symbol section was subsequently removed by bc3b35f97; it is not a current public API compatibility guarantee.

Compatibility and adjacent work

Readers that ignore unknown XML elements remain compatible. No new algorithm, stored metadata format, object migration or checksum bypass was introduced. The UploadPart repair and completion validation are separate changes, also included in Server 20260903. In particular, CRC64NVME + COMPOSITE is rejected, not silently normalized.

4.22 - When the Total Is Unknown: Folder Download Progress

Historical scope: this is the Console 2.2.0 progress repair, included in Server 20260903’s embedded Console. Console 2.4.1 subsequently added file streaming and native browser handoff, eliminating the complete JavaScript ZIP Blob. Transport descriptions and follow-ups in the original PRD below refer to the 2.2.0 design point.

Status: Shipped in SILO Console 2.2.0 (16960f7ab); the server embeds it since its Console pin was updated (edc8be6ed) · Priority: P1 · Owner: pgsty/silo-console · Related issue: pgsty/silo#62 · PRD review: Claude Fable 5 (xhigh) — APPROVE · Implementation review: Claude Fable 5 (xhigh), 2026-08-23 — APPROVE, no P0/P1/P2 findings

SILO Console shows NaN% in Downloads / Uploads while downloading a folder. The ZIP normally keeps streaming and the stored objects are intact, but the progress bar has crossed from “unknown” into an invalid determinate state. Users see a full-looking bar, assume the transfer failed or finished, and retry it.

The proposed repair is intentionally narrow:

A download may enter determinate mode only when it has a finite, positive total measured in bytes applicable to that response. Without such a total, it remains indeterminate until completion, failure, or cancellation.

The server keeps streaming ZIPs. Ordinary files keep their percentages. The frontend gains one safe calculation boundary, reuses its existing indeterminate renderer, and closes one missing cancellation transition. This record defines why that is both sufficient and the smallest truthful fix.

The observed failure

The defect was observed in the then-current silo-console v2.1.1, which is embedded by Silo RELEASE.2026-08-06T00-00-00Z.

Reproduction:

  1. Put several objects below a prefix such as folder/.
  2. Stay in the parent listing, select folder/, and click Download.
  3. Open Downloads / Uploads before the transfer finishes.
  4. The row displays NaN%; the ZIP request continues.

The runtime check used a prefix containing about 88.7 MiB and throttled Chromium to preserve the observation window. Two independent downloads produced the same NaN% state.

This is a frontend correctness bug. It is not evidence of corrupted objects, an altered disk format, or a failed S3 GET.

What is actually happening

The visible NaN% is the end of a contract mismatch across three layers.

A prefix has no object size

S3 folders are common prefixes, not stored directory objects. In the listing model, a prefix ends in / and carries size=0. The Console already renders that size as -, correctly treating it as not applicable.

The generated API model marks size as omitempty, so logical zeroes are absent from listing JSON. The single-selection thunk nevertheless passes object.size straight into the download helper: a prefix or zero-byte object therefore supplies undefined at runtime (while synthetic prefix records may supply 0). Neither value is a valid denominator.

A streamed ZIP has no known wire length

The server recognizes the trailing /, recursively lists the objects, then connects a zip.Writer to an io.Pipe. Objects are read, deflated, and copied to the HTTP response as the archive is produced.

That behavior is desirable: the server can send the first bytes without holding the complete archive in memory or on disk. Its consequence is equally deliberate: the final compressed byte length does not exist when headers are sent, so the response has Content-Type: application/zip and a filename, but no Content-Length.

The sum of source object sizes is not a substitute. Source sizes are uncompressed bytes; ProgressEvent.loaded counts response bytes after ZIP compression and framing. They are different units.

A progress event does not imply a computable percentage

The client currently computes every event as:

Math.round((event.loaded / fileSize) * 100)

For a prefix, the denominator is zero or absent. Depending on the value and event, JavaScript produces NaN (loaded / undefined or 0 / 0) or Infinity (positive bytes divided by zero).

The progress callback then writes that non-finite value into Redux and sets waitingForFile=false. That second operation is the decisive state error: the task leaves the existing indeterminate branch merely because an event arrived, not because the event contained a usable total. The determinate progress component receives the invalid value and renders an invalid label.

The complete chain is:

common prefix: size = 0
        |
        v
download(..., fileSize = 0)
        |
        v
streamed deflated ZIP, no Content-Length
        |
        v
event.loaded / 0 => NaN or Infinity
        |
        v
invalid percentage enters Redux; waitingForFile becomes false
        |
        v
determinate ProgressBar renders NaN%

Ordinary non-empty files avoid the defect because the server can stat the object, sets Content-Length, and the list size is positive. If the browser emits a progress event for an empty response, a zero-byte file reaches the same arithmetic boundary as a prefix even though it is a real object; it therefore belongs in the regression contract.

Product contract

The UI needs one honest distinction:

  • Determinate means both transferred bytes and total bytes are known in the same unit.
  • Indeterminate means the request is active but the total is unknown.

This yields four load-bearing invariants:

determinate  => total is finite and total > 0
determinate  => percentage is finite and 0 <= percentage <= 100
unknown total => indeterminate
terminal state => not indeterminate

These invariants are more general than objectPath.endsWith("/"): they cover prefixes, zero-byte files, malformed metadata, and any future unknown-length response without inventing object-type exceptions.

Goals and non-goals

Goals

  1. A folder download never displays NaN%, Infinity%, or a fabricated percentage.
  2. Unknown-length transfers use the existing indeterminate animation.
  3. Known-length ordinary files retain their current percentage behavior.
  4. Completion, failure, and cancellation always leave indeterminate mode.
  5. A zero-byte file never produces a non-finite percentage and still reaches success.
  6. No non-finite or out-of-range download percentage enters Redux.
  7. The fix can ship in Console first and then be consumed by Silo as a dependency update.

Non-goals

  • Do not pre-generate or buffer a complete ZIP on the server.
  • Do not use the sum of uncompressed object sizes as network progress.
  • Do not redesign the entire Object Manager state model.
  • Do not route folders through the current immediately-completing BrowserDownload path.
  • Do not solve the browser memory cost of XMLHttpRequest.responseType="blob" here.
  • Do not change whether a cancelled row remains visible until the user clears it.
  • Do not redesign mid-stream ZIP error signaling after HTTP headers have been sent.
  • Do not modify the S3 API, Console API, object layout, or archive contents.

Those are legitimate follow-ups, but coupling them to this defect would enlarge risk without being necessary to restore truthful progress.

The decision

The minimum production repair has four parts.

D1. Calculate only from a valid total

Add a small pure function, separate from DOM and Redux side effects:

type DownloadProgressEvent = Pick<
  ProgressEvent,
  "loaded" | "lengthComputable" | "total"
>;

export const calculateDownloadPercent = (
  event: DownloadProgressEvent,
  objectSize: number,
): number | null => {
  let total: number | null = null;

  if (Number.isFinite(objectSize) && objectSize > 0) {
    total = objectSize;
  } else if (
    event.lengthComputable &&
    Number.isFinite(event.total) &&
    event.total > 0
  ) {
    total = event.total;
  }

  if (
    total === null ||
    !Number.isFinite(event.loaded) ||
    event.loaded < 0
  ) {
    return null;
  }

  return Math.min(
    100,
    Math.max(0, Math.round((event.loaded / total) * 100)),
  );
};

The source priority preserves compatibility:

  1. A finite positive objectSize retains the current ordinary-file calculation.
  2. If object size is unavailable but the browser declares the response length computable and supplies a finite positive event.total, use it.
  3. Otherwise return null: no truthful percentage exists yet.

The helper’s output contract is complete: either null, or a finite number in [0,100].

D2. Keep unknown totals indeterminate

Change the XHR handler to dispatch only a real percentage:

req.addEventListener("progress", (event) => {
  const percent = calculateDownloadPercent(event, fileSize);

  if (percent !== null) {
    progressCallback(percent);
  }

  // No valid total: preserve waitingForFile=true so the existing UI remains
  // indeterminate instead of manufacturing a determinate value.
});

Download rows already start with waitingForFile=true, and ObjectHandled already renders that state with variant="indeterminate". There is no need to widen Redux to number | null, add another boolean, or change MDS.

When the first valid percentage arrives, the existing updateProgress action stores it and sets waitingForFile=false. When no valid percentage ever arrives, the row remains indeterminate until a terminal action.

D3. Make cancellation terminal

Completion and failure already clear waitingForFile. Cancellation does not. Add the missing transition in cancelObjectInList:

item.waitingForFile = false;

Without that line, the repaired prefix download would remain in the indeterminate rendering branch after abort, masking the Cancelled state. The row continues to follow the current product behavior: it remains as a cancelled record and can be removed manually. Automatic removal is not part of this change.

There is one event-order guard at the XHR boundary as well. abort() first produces readystatechange(DONE, status=0) and only then the abort event; without a status-zero return, the generic DONE branch marks the request failed before onabort can mark it cancelled. DONE/status zero is therefore left to the dedicated onerror or onabort handler, and onabort removes the stored request reference.

D4. Normalize an omitted zero-byte size

The single-selection thunk passes object.size || 0, matching the other download entry point. This restores the API model’s omitted logical zero before the helper checks Blob.size === fileSize, so an HTTP 200 zero-byte object completes at 100% instead of being reported as incomplete.

D5. Keep the server stream unchanged

The folder handler continues to generate a deflated ZIP through io.Pipe and omit Content-Length. No API, archive, storage, or resource-management contract changes.

State machine

State waitingForFile percentage Terminal flag Rendering
Queued / no valid progress yet true 0 none indeterminate
Unknown-total transfer true 0 none indeterminate
Known-total transfer false 0..100 none determinate percentage
Completed false 100 done=true success
Failed false last value failed=true, done=true error
Cancelled false 0 cancelled=true, done=true cancelled

The state does not move back from determinate to indeterminate. If a later event lacks a valid total after a valid percentage was observed, the handler simply retains the last valid value.

Failed and Cancelled both set done=true in the existing reducers. ObjectHandled uses done to change its close button from “abort request” to “remove record”; this repair preserves that behavior. The cancelled Redux value remains 0, while the existing ProgressBarWrapper renders a full orange terminal bar with a Cancelled label because ready=true. That established presentation is not part of this repair.

waitingForFile is not the ideal long-term name for “no computable progress.” Renaming it or replacing the booleans with a discriminated union would improve the model, but that is a separate refactor. In this repair, the field already expresses and renders the required state, so reusing it minimizes compatibility risk.

Why this is sufficient

The repair closes the bug by cases.

Ordinary non-empty file

objectSize > 0, so the helper uses the same denominator as today. The result is finite and clamped, updateProgress enters determinate mode, and completion still sets 100%.

Current streamed folder

objectSize is normalized to 0, while lengthComputable=false and event.total=0. The helper returns null; no invalid action is dispatched, so the row remains indeterminate. Completion sets waitingForFile=false, percentage=100, and done=true.

Future response with a real length

If a proxy or later server implementation provides a trustworthy response total, lengthComputable=true and event.total>0. The same code automatically produces a real percentage without another product change.

Zero-byte file

The omitted listing size is normalized to zero, and both totals are then zero, so an intermediate percentage is mathematically undefined. The row stays indeterminate for its usually brief lifetime; the zero-byte Blob now equals the normalized expected size, and the successful response transitions directly to 100%. 0/0 is never evaluated.

Failure and cancellation

Failure already exits indeterminate. The added cancellation transition does the same on abort. No terminal row can continue to look active merely because its total was unknown.

Mathematically, division occurs only when total belongs to (0, +infinity). The result is then clamped to [0,100]. Therefore neither NaN nor Infinity can cross the calculation boundary into Redux or the determinate renderer.

Rejected alternatives

Buffer the ZIP to obtain Content-Length

The server could generate the complete archive in memory or a temporary file, measure it, and then send it. That would provide an exact wire total, but at the cost of memory or disk pressure, delayed first byte, cleanup complexity, and worse concurrent-download behavior. An observability defect does not justify discarding streaming.

Sum the objects under the prefix

That sum is uncompressed logical data. event.loaded measures compressed response bytes plus ZIP framing. The units differ, so the bar could stop below 100%, exceed 100%, or move according to compression ratio rather than transfer completion. Reject.

Convert invalid progress to 0%

This hides the string but lies about the state: determinate 0% means the total is known and no portion has transferred. Users would still interpret the transfer as stalled. Unknown must remain unknown.

Special-case paths ending in /

That fixes the reported prefix but misses a real zero-byte object, invalid metadata, and other unknown-length responses. The correct boundary is denominator capability, not object type.

Send folders through BrowserDownload

The current large-file path creates an anchor and immediately calls the completion callback after clicking it. It cannot report true completion, console-managed cancellation, or a subsequent HTTP failure. It may be the basis of a later streaming-download design, but today it would replace one lie with another.

Sanitize inside ProgressBar

A generic component guard could be useful defense in depth, but it would leave invalid data in Redux and hide the broken state transition from every other consumer. The primary repair belongs where progress becomes application state.

Introduce percentage: number | null now

A discriminated progress state would be cleaner than the current booleans if the Object Manager were being redesigned. Adding null while retaining waitingForFile, done, failed, and cancelled would instead create more contradictory combinations. Removing the old fields is larger than this bug requires. Reuse the already-rendered indeterminate state now; redesign it separately.

Requirements and acceptance

Functional requirements

  • FR1: An unknown total keeps the task indeterminate.
  • FR2: A finite positive object size preserves ordinary-file percentages.
  • FR3: A finite positive event.total is a fallback only when lengthComputable=true.
  • FR4: Every dispatched percentage is finite and within [0,100].
  • FR5: A zero-byte file never displays non-finite progress and reaches success.
  • FR6: Completion, failure, and cancellation leave indeterminate mode.
  • FR7: Versioned objects, anonymous downloads, previews, and long-filename entry points retain their existing call contract.

Non-functional requirements

  • No new server CPU, memory, disk-buffer, or request cost.
  • No new frontend dependency or build step.
  • No change to the S3 API, Console API, ZIP content, or stored objects.
  • The calculation must be testable without a DOM or live store.
  • TypeScript typecheck and the production frontend build must pass.

Acceptance criteria

  1. While a folder ZIP without Content-Length is active, its row shows an indeterminate animation and no percentage text.
  2. On successful completion, the row reports success/100% and the ZIP can be opened.
  3. A normal non-empty file continues to show finite determinate progress and completes at 100%.
  4. A zero-byte file never shows NaN% or Infinity% and completes successfully.
  5. Cancelling an unknown-total download aborts the request and shows Cancelled, not an active animation.
  6. No download path can place a non-finite or out-of-range percentage in Redux.

Test plan

Pure calculation matrix

Use the existing @playwright/test runner for the pure module rather than adding a test framework. This needs one config-only addition in web-app/playwright.config.ts: a dependency-free unit project, for example with testMatch: /.*\.unit\.ts/. The existing chromium project depends on the auth setup against a live Console at localhost:9090; pure calculation and reducer tests must not be gated by that environment. No new dependency is introduced.

Case loaded objectSize lengthComputable event.total Expected
Ordinary file, halfway 50 100 false 0 50
Common prefix 1024 0 false 0 null
Initial zero over zero 0 0 false 0 null
Response-total fallback 50 0 true 200 25
Zero total is unusable 0 0 true 0 null
Loaded exceeds total 150 100 true 100 100
Invalid object size 10 NaN false 0 null
Omitted zero size 10 undefined false 0 null
Invalid response total 10 0 true Infinity null
Negative loaded -1 100 true 100 null

State tests

Cover the transition contract directly:

  1. A new download starts with waitingForFile=true.
  2. No valid progress action means it remains indeterminate.
  3. Valid progress produces a finite value and waitingForFile=false.
  4. Complete produces done=true, waitingForFile=false, percentage=100.
  5. Failure produces failed=true, done=true, waitingForFile=false.
  6. Cancel produces cancelled=true, done=true, waitingForFile=false, percentage=0.

Browser regression

Use the real Console test instance and Chromium:

  1. Create a temporary bucket with several objects below folder/.
  2. Select the prefix from its parent and start the download.
  3. Apply CDP download throttling so the intermediate state is observable. Throttled runs must raise the default 30-second test timeout with test.setTimeout.
  4. Open Downloads / Uploads and verify that the row exists, has no percentage label, and contains neither NaN% nor Infinity%.
  5. Cancel it and verify the Cancelled terminal state.
  6. Restore network conditions in finally.
  7. Download again without throttling, wait for the browser download, and verify the ZIP.
  8. Repeat the relevant assertions for one ordinary non-empty file and one zero-byte file.
  9. Remove the bucket, objects, downloads, and temporary files in teardown.

The current Playwright project is Chromium-only, so CDP is an acceptable test mechanism. If Firefox or WebKit projects are later enabled, keep the pure and state tests cross-browser and gate only the throttled observation behind the Chromium project.

Implementation boundary

Expected Console changes:

  1. Add downloadProgress.ts containing the pure calculation.
  2. Change Objects/utils.ts to dispatch only a non-null percentage, let status-zero terminal events reach their dedicated handlers, and clean up an aborted request.
  3. Normalize omitted zero sizes in the single-selection thunk.
  4. Change cancelObjectInList to clear waitingForFile.
  5. Add calculation, state, and browser regression coverage using existing dependencies, with a dependency-free unit project in playwright.config.ts.

Expected unchanged code and contracts:

  • The Go folder-download handler and its streaming ZIP.
  • ObjectHandled, ProgressBarWrapper, and MDS.
  • IFileItem.percentage: number and the existing thunk callback types.
  • S3 and Console API routes.
  • Stored object and archive formats.

Delivery and rollback

The fix belongs in pgsty/silo-console, not the Silo server repository where the issue was reported.

Delivery order:

  1. Transfer or cross-reference issue #62 to pgsty/silo-console.
  2. Implement the bounded Console change.
  3. Pass typecheck, production build, pure/state tests, and real browser regression.
  4. Publish a new Console release.
  5. Update Silo’s pinned Console pseudo-version or release dependency.
  6. Build a Silo candidate and repeat folder, ordinary-file, zero-byte, cancel, and ZIP-integrity checks.
  7. Publish Silo and record both affected and fixed versions on the issue.

There is no data migration. If the frontend change regresses, Silo can roll back only the Console dependency; server data and API behavior remain compatible.

Definition of done

  • The calculation returns only null or a finite [0,100] number.
  • Active unknown-total folder downloads render indeterminate.
  • Ordinary files retain determinate progress.
  • Zero-byte files never render invalid progress.
  • Complete, failed, and cancelled rows all leave indeterminate mode.
  • The streamed ZIP and server response contract remain unchanged.
  • Typecheck, production build, and automated regressions pass locally.
  • Console 2.2.0 is published.
  • Server 20260903 includes the updated Console; see its release evidence.

Follow-up work

Five adjacent improvements deserve separate design records:

  1. Stream large folder downloads directly to the browser or filesystem instead of holding the full Blob in memory.
  2. Replace the Object Manager’s boolean combination with a discriminated progress/terminal state.
  3. Improve end-to-end integrity and error signaling for ZIP failures after headers have been sent.
  4. Add a generic non-finite-value guard to shared progress components as defense in depth.
  5. Repair the pre-existing Blob JSON error decoder and request-trace cleanup on HTTP failure paths.

None is required to stop the current UI from lying. The next maintenance iteration should first restore the smallest honest contract: known totals get percentages; unknown totals remain unknown.

Later implementation and the original design

The Console 2.2.0 integration also made size an always-present JSON field and repaired ZIP error propagation: unreadable entries are no longer silently skipped; errors before output can return 500 and errors afterward interrupt the stream. The no-API/resource-management-change statements above describe only the original progress-calculation patch, not the whole release.

Current single-folder downloads are handed to the browser. A completed Console row records the handoff, not completion of all bytes; track and cancel the transfer in the browser download manager. Multi-selection file-writer/native handoff paths are described in Console 2.4.1. Size normalization remains defensive support for old responses, not evidence that the current model omits zero values.

4.23 - A ListObjects Shortcut Must Not Turn a Missing Bucket into an Empty One

Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.

This document records the problem analysis, design discussion, and repair decision for SILO #32 and PR #37.

Status on 2026-08-26: PR #37 was updated to the DCO-signed head e9c5340be, formally approved, and merged as 49c8aeac4; #32 closed automatically. DCO, VulnCheck, and all six Go CI jobs passed on the exact PR head; the post-merge main VulnCheck and all six Go CI jobs also passed. No tagged release, package, container image, deployment, or production endpoint has yet been verified to contain the repair.
Scope: verify bucket existence only for three listing shortcuts that bypass storage; do not restore the generic checkBucketExist, change the normal listing path, or introduce an existence cache.
Release boundary: local commit, push, remote CI, merge, tag, package, container image, deployment, and production verification are independent gates.

Too Long; Didn’t Read (TL;DR)

The problem is real and worth fixing. A normal ListObjects, ListObjectsV2, or ListObjectVersions request against a missing bucket reaches storage and receives BucketNotFound. Three inputs, however, return early:

  • a marker outside the prefix;
  • max-keys=0;
  • a prefix beginning with /, including the Prefix="/" boto3 reproduction from #32.

Those branches return io.EOF directly. The caller treats EOF as a successful end of listing, so the client receives an empty 200 rather than S3’s 404 NoSuchBucket. The identity of the same missing resource changes from an error to success solely because the selection parameters differ. That breaks S3 compatibility and blocks a real user’s upgrade from the pre-regression release.

The repair must not put an expensive bucket check back in every listing. The selected design replaces only the three bare io.EOF returns with a small helper. The helper calls GetBucketInfo once: it returns the real error if the bucket is absent or cannot be confirmed, and preserves io.EOF when the bucket exists. The normal listing hot path is untouched. Only requests that would otherwise exit before storage pay the extra peer-and-disk fan-out.

That decision has now been executed: the strengthened repair passed local review, the exact PR head passed every remote check, and the expected-head-guarded merge entered a green main.

What is the problem?

One API exposes two bucket-existence semantics

#32 reproduces the defect by calling the following against a missing bucket:

s3.list_objects(Bucket="missing-bucket", Prefix="/")

AWS S3 raises NoSuchBucket; SILO returns a successful empty listing. The difference is not in authentication, routing, or XML serialization. It comes from the object-layer listPath control flow:

regular prefix
  -> enter listMerged
  -> consult storage
  -> missing volume/bucket becomes BucketNotFound
  -> HTTP 404 NoSuchBucket

shortcut input
  -> listPath returns io.EOF early
  -> storage is never consulted
  -> the caller treats EOF as normal completion
  -> HTTP 200 with an empty listing

/ is not the only trigger:

Shortcut condition Why the result must be empty Defect before the repair
Marker does not begin with the prefix The implementation does not scan this disjoint range Returns EOF without confirming the bucket
max-keys=0 The caller asks for zero keys Incorrectly equates “zero results” with “valid resource”
Prefix begins with / SILO’s flat key space produces no entries for this form The filter short-circuits before bucket identity

For an existing bucket, returning an empty listing from these branches is a reasonable optimization. For a missing bucket, the same EOF masks the resource error that should take precedence.

The regression has a known origin

The reporter confirmed correct behavior in RELEASE.2024-01-29T03-56-32Z and the regression beginning with RELEASE.2024-01-31T20-20-33Z. The corresponding upstream change is minio/minio#18917 / 80ca12008. It removed GetBucketInfo from generic argument checks and relied on actual Put, List, and Multipart storage operations to expose a missing bucket.

That optimization works on normal paths but leaves a gap: an early-return path never reaches the storage operation that is now responsible for producing the error. #32 does not require a broad rollback of the upstream optimization. It repairs the overlooked control-flow exits.

Why fix it?

The S3 contract explicitly requires NoSuchBucket

Both AWS ListObjects and ListObjectsV2 define NoSuchBucket as HTTP 404 when the specified bucket does not exist. prefix, marker, start-after, and max-keys select listing results; they must not turn a missing bucket identity into a successful request.

ListObjectVersions shares the same object-layer listing engine. Giving V1, V2, and version listings the same existence behavior on the same shortcut inputs prevents the three public APIs from diverging further.

An empty 200 changes client decisions

An empty 200 and a 404 are not interchangeable presentation details:

  • 404 tells provisioning or test code to create the bucket, fix configuration, or stop;
  • an empty 200 asserts that the bucket exists but has no matching objects;
  • SDKs, synchronization tools, and integration tests continue down different branches;
  • a test using SILO as an S3 substitute can pass locally and fail against AWS.

#32 also establishes a direct upgrade impact: an application relying on the older correct behavior cannot upgrade past the regression. The repair restores both S3 parity and upgrade compatibility.

The repair surface is narrow and testable

The bug is confined to three adjacent early returns. It does not involve object data, metadata formats, sorting, pagination-token encoding, permissions, or wire schemas. A very small production change can be pinned down with object-layer and HTTP-level contracts, so the benefit clearly exceeds the implementation risk.

Why not restore the global check?

Upstream did not remove generic GetBucketInfo as incidental cleanup. The motivation for #18917 states that checking the bucket before every Put, List, and Multipart operation fans out across servers; even after vectorization, the cost becomes visible beyond 100 nodes.

In current SILO, erasureServerPools.GetBucketInfo calls S3PeerSys.GetBucketInfo. That operation concurrently asks every peer and reduces quorum per pool, while each peer checks its local bucket state. It is not a cheap in-memory map lookup.

Two extremes are therefore unacceptable:

  • never check: keep the incorrect empty 200;
  • check before every List: restore semantics while undoing a critical large-cluster optimization.

The actual design question is whether the check can be confined to branches that never touch storage and therefore cannot discover the missing bucket naturally. It can.

How is it fixed?

Replace only three bare EOF returns

In cmd/metacache-server-pool.go, each shortcut previously executed:

return entries, io.EOF

It now executes:

return entries, z.listPathShortcutEOF(ctx, o.Bucket)

The helper has only two classes of outcome:

func (z *erasureServerPools) listPathShortcutEOF(ctx context.Context, bucket string) error {
    if _, err := z.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
        return err
    }
    return io.EOF
}
  • existing bucket: preserve the previous empty-list behavior;
  • missing bucket: pass BucketNotFound into the existing error mapping, producing HTTP 404 NoSuchBucket;
  • state cannot be confirmed: propagate quorum, offline, timeout, or context errors instead of fabricating success.

The normal listMerged, metacache scan, sorting, pagination, and response-generation paths do not change.

Why the helper belongs here

The check must sit next to the shortcut for three reasons:

  1. only this layer knows that it is about to bypass every storage access;
  2. moving it into generic argument validation charges every call;
  3. moving it into the scan layer cannot help because these branches never scan.

The name intentionally states the boundary. This is not a new generic checkBucketExist; it restores missing existence semantics immediately before a shortcut returns EOF.

Do not add a cache

A bucket-existence cache could reduce fan-out but immediately creates invalidation questions for create, delete, site replication, recovery, and expiry. Adding a second source of truth for three low-frequency shortcuts costs more complexity and consistency risk than it saves.

The selected implementation uses the existing GetBucketInfo source of truth. If future telemetry shows that large clusters receive frequent max-keys=0, slash-prefix, or disjoint-marker probes, the project can evaluate a dedicated metadata fast path, rate limiting, or a carefully invalidated cache using real data rather than speculative machinery in this compatibility patch.

Test and review evidence

Object-layer contract

The object-layer test runs against single-drive and multi-drive erasure setups and exercises four inputs:

  • slash-prefixed prefix;
  • zero limit;
  • marker outside prefix;
  • a regular prefix as a control that still receives the error naturally from storage.

Each case covers ListObjects, ListObjectsV2, and ListObjectVersions, using the typed isErrBucketNotFound predicate rather than brittle English error-string comparison.

HTTP contract

The handler test sends genuine signed requests for all three public APIs:

API Request shape Assertion
ListObjects GET /missing-bucket?prefix=/ HTTP 404 and XML code NoSuchBucket
ListObjectsV2 Add list-type=2 HTTP 404 and XML code NoSuchBucket
ListObjectVersions Add versions HTTP 404 and XML code NoSuchBucket

The HTTP test uses the real slash-prefix reproduction from #32. The other two shortcuts are enumerated at the object layer. This proves final wire behavior without repeating the full matrix in the slower handler fixture.

Local quality gates

The improved local commit passed:

go test ./cmd -count=1
the new object-layer and HTTP regressions (10 subcases)
focused go test -race
related existing listing tests
CGO_ENABLED=0 go build ./...
go vet ./...
CI-scope gofmt and git diff --check
post-commit focused regression rerun

The full local cmd test completed in 116.215 seconds. An independent local Claude Code review used the Fable model at Max effort to inspect the exact tree, call paths, error mapping, tests, performance boundary, and this decision. Its verdict was GO, with no mandatory pre-merge change.

The DCO-signed PR head e9c5340be then passed eight remote checks: DCO, VulnCheck, and six jobs in Go CI. After merge, the resulting main commit 49c8aeac4 independently passed VulnCheck and all six Go CI jobs. The slowest checks were PR cross-compile at 9 minutes 47 seconds and post-merge cross-compile at 9 minutes 30 seconds.

Can it introduce new problems?

Shortcut requests now fan out across the cluster

This is the most important and deliberately accepted cost. A shortcut on an existing bucket used to be little more than a local branch; it now calls GetBucketInfo. Directional local microbenchmarks observed:

Path Observed magnitude
Shortcut before the repair about 0.55 μs, 7 allocations
Repaired single-drive shortcut about 7.8–8.1 μs, 45–47 allocations
Repaired 32-drive shortcut about 70–81 μs, 977 allocations
Normal 32-drive listing about 0.95 ms

These numbers show local relative cost only; they are not a latency prediction for a 100+ node deployment. Real distributed execution adds peer networks, quorum, and slowest-node tail latency, potentially making the gap much larger. That is precisely why the check must not expand into the normal listing path.

The risk concentrates in malformed or probe-style traffic. A misconfigured client polling max-keys=0, a slash prefix, or disjoint markers at high frequency can amplify what was a cheap request into peer-and-disk work. After merge, the actual frequency of these inputs should be observed through S3 traces or metrics; rate limiting or optimization should follow evidence.

A degraded cluster exposes more real errors

Previously, a shortcut could return an empty 200 while peers were offline or bucket quorum was unavailable because it never consulted cluster state. The repair can return quorum, timeout, or service errors in those conditions.

That is more honest behavior, not an availability regression: if the server cannot establish that the bucket exists, it must not assert a valid empty bucket. Clients depending on unconditional empty success will nevertheless observe a behavior change.

Bucket create/delete races are not linearizable

GetBucketInfo and returning the empty result are two actions. The bucket can be deleted immediately after the check, or created immediately after a missing-bucket result is formed. This patch does not and should not add a transaction spanning bucket lifecycle to a listing shortcut.

This is the same concurrency class as other APIs that validate a resource before acting. The repair guarantees that the request no longer succeeds with no existence evidence at all; it does not promise a cross-node, cross-lifecycle linearizable snapshot of an empty listing.

Clients relying on the bug will receive 404

Some clients may have adopted the missing bucket’s empty 200 as fact. They will now enter an error branch. This is a visible compatibility change, but it restores the documented S3 contract and the pre-regression behavior. Preserving the bug merely transfers upgrade cost to clients that correctly rely on 404.

Two adjacent edges remain out of scope

The adversarial review recorded two non-blocking P3 boundaries:

  1. When resuming a metacache continuation, the c.fileNotFound branch still returns bare io.EOF. A stale or crafted continuation token used after bucket deletion could theoretically receive an empty 200. Adding GetBucketInfo there would affect normal continuation traffic and needs a separate performance and error-precedence design.
  2. Some V1 and version-list marker/prefix combinations return NotImplemented during HTTP handler validation before reaching the object layer; the V2 start-after route can reach it. This patch fixes storage shortcuts masking a missing bucket; it does not redefine precedence between malformed parameters and resource errors.

Neither blocks merge. The first is outside #32’s ordinary initial-list reproduction; the second is inherited handler behavior. Recording them prevents “all three shortcuts are covered” from being overstated as byte-for-byte AWS parity for every possible parameter combination.

Alternatives considered

Keep upstream behavior

This has zero performance change and minimizes fork divergence. It also keeps a documented S3 incompatibility, a regression with a known release boundary, and a misleading result when SILO is used as an integration-test substitute. For a narrow and well-tested compatibility repair, that tradeoff is no longer justified.

Restore generic checkBucketExist

This covers every path at once but reintroduces peer fan-out into every Put, List, and Multipart operation, directly undoing the large-cluster optimization from #18917. The cost is disproportionate and the option is rejected.

Fix only Prefix="/"

That passes the single issue reproduction but leaves the same root defect in max-keys=0 and marker-outside-prefix. The branches are adjacent and share the same semantics, so one helper is simpler and less likely to regress.

Add a bucket-existence cache

This makes shortcuts cheaper but requires semantics for create, delete, replication, recovery, and stale TTL windows. There is no telemetry showing enough shortcut traffic to justify that complexity, so it is not selected.

Complexity and cost-benefit

Dimension Assessment Rationale
Production-code complexity Low Three call sites and a seven-line helper; no new state, dependency, or format
Test complexity Low to medium V1, V2, versions, three shortcuts, a control, and HTTP mapping all need coverage
Normal-path risk Very low No check is added to the listMerged hot path
Shortcut runtime cost Materially higher A local EOF becomes cluster-wide GetBucketInfo
Compatibility value High Restores 404 NoSuchBucket, pre-regression behavior, and S3 test fidelity
Operational complexity Low No migration, configuration, feature flag, cache, or cross-repository dependency

The overall cost-benefit is favorable. The reason is not that GetBucketInfo is cheap—it is not—but that its cost is strictly limited to three shortcuts that otherwise cannot discover the missing bucket. A narrow performance cost in exchange for explicit protocol correctness is better than either a global rollback or indefinitely preserving the incorrect behavior.

Acceptance decision and remaining gates

The final decision was: accept and merge the strengthened PR #37 revision without expanding the production scope.

The accepted sequence was:

  1. replace the old fork head with the current-main, DCO-signed revision while preserving Jason Lin as a co-author;
  2. retain typed error predicates, V1/V2/version-list object-layer coverage, and HTTP-level 404 / NoSuchBucket assertions;
  3. update the PR description with the shortcut fan-out cost and unchanged normal-path boundary;
  4. approve the fork workflows and require all eight reported checks to pass on exact head e9c5340be;
  5. submit a formal approving review against that head;
  6. merge with an expected-head guard, producing 49c8aeac4, automatically close #32, and require the resulting main Go CI and VulnCheck to pass independently.

No cache, feature flag, additional abstraction, or continuation-token redesign was required. High-frequency shortcut traffic and large-cluster tail latency remain observability follow-ups, not reasons for speculative code expansion.

Repository integration is complete. A tag, package, docker.io/pgsty/silo image, deployment, and real S3-client verification must still complete before the repair can be described as delivered to users.

Conclusion

The issue is not merely “a slash prefix reports the wrong error.” The listing engine uses io.EOF to mean two different things: an empty result from an existing bucket and an early exit that never established whether the bucket exists. Removing generic existence checks for large-cluster performance was a sound upstream optimization, but the shortcuts violate its premise that a real storage operation will naturally surface a missing bucket.

The selected repair restores that premise by calling the existing GetBucketInfo only at three storage-bypassing exits. It makes those requests more expensive and exposes real errors on degraded clusters; both are explicit costs. In return, SILO restores S3’s 404 semantics, upgrade compatibility, and test fidelity while preserving the upstream optimization on the normal listing hot path.

This worthwhile, controlled compatibility fix is now merged and green on main; the repair is included in Server 20260903; production deployment remains installation-specific.

4.24 - Read-Only Checksum Audit and Reliable CLI Output

This is the design and implementation record for MCLI’s read-only checksum verification workflow and pgsty/mc#5, the non-TTY output defect found during release review.

Status: shipped in the final mcli 20260903 release. The command merged to main through pull requests #8 and #13, is exercised against a real SILO server in hosted CI, and pgsty/mc#5 is closed. Server 20260903 images bundle mcli 20260903, including this command.
Owner: pgsty/mc.
Tracking: pgsty/mc#5.
Safety boundary: verification is read-only; repair is not part of this command.

Too Long; Didn’t Read (TL;DR)

Historical CopyObject implementations could calculate a stored additional checksum over transformed storage bytes instead of the logical bytes returned by S3. mcli checksum verify inventories objects and independently streams the logical body through the recorded algorithm. Each candidate becomes MATCH, MISMATCH, NO_CHECKSUM, WOULD_VERIFY (dry run), one of ten UNKNOWN_* classifications, or one of three SKIPPED_* results.

The first implementation worked in a terminal but printed nothing when stdout was redirected. MCLI automatically marked non-TTY execution as quiet to disable progress UI, and the new command accidentally treated that internal state as a user request to suppress audit records. The repair separates semantic output from progress suppression without changing global quiet behavior or enabling progress bars in CI.

Command and scope

mcli checksum verify ALIAS/BUCKET/OBJECT
mcli checksum verify --recursive ALIAS/BUCKET[/PREFIX]
mcli checksum verify --manifest candidates.jsonl ALIAS

Version one supports CRC32, CRC32C, CRC64NVME, SHA1, and SHA256 checksums marked as FULL_OBJECT. It can select one object, an exact VersionID, current objects under a prefix, all versions, or exact entries from a JSON Lines manifest. It also supports SSE-C key mappings, time and size filters, dry-run estimation, bounded workers, download limits, JSON output, and an optional JSON Lines report.

It does not verify COMPOSITE checksums, infer type from an ETag, inspect xl.meta, identify the historical writer with certainty, or repair metadata. The endpoint must report the checksum type (x-amz-checksum-type) alongside the checksum; on one that does not, every checksummed object is classified UNKNOWN_CHECKSUM_TYPE rather than guessed at.

Read-only data path

For every selected object, MCLI:

  1. sends HEAD with checksum mode enabled and retains every supported checksum plus ChecksumType;
  2. rejects unsupported or ambiguous states as UNKNOWN_* instead of guessing;
  3. streams GET logical bytes through bounded hashers without writing the body to disk;
  4. uses VersionID pinning, or If-Match plus a second HEAD for mutable unversioned/null objects;
  5. compares independently calculated values with the stored values.

The S3 boundary allows LIST, HEAD, and GET only. Tests fail if a write method reaches the mock endpoint.

Result and exit contract

Every candidate produces one stable result:

Result Meaning
MATCH Every supported stored checksum matches the returned logical bytes
MISMATCH At least one stored checksum differs
NO_CHECKSUM No additional checksum exists; the body is not read
WOULD_VERIFY Dry-run found a supported full-object checksum
UNKNOWN_* MCLI cannot make a reliable statement
SKIPPED_* A filter intentionally excluded the object

The summary carries objects, a verified count, the count of every result status, and incomplete. verified is MATCH plus MISMATCH: the only results that actually streamed a body through a hasher. A run that enumerated many objects and verified none is visible as such.

--fail-on accepts mismatch, unknown, no-checksum, any, or none. The default any returns exit 1 for mismatches and incomplete verification. no-checksum returns exit 1 when any object carries no checksum or when nothing was verified at all, so an empty prefix or a stale manifest cannot pass as a clean audit. Dry-run does not apply --fail-on. Argument, authentication, enumeration, and report-write failures remain command failures rather than object classifications.

In particular, SKIPPED_TOO_LARGE makes the default any return exit 1 because the size cap leaves the audit incomplete. Time-filter and delete-marker skips do not fail by themselves.

Output and automation contract

Object records and the final summary are semantic output:

  • Unless the caller explicitly sets --quiet, -q, or MC_QUIET=true, stdout receives every object record and the final summary in both TTY and non-TTY execution.
  • Non-TTY --json emits exactly one compact JSON value per line. TTY JSON keeps MCLI’s existing pretty presentation.
  • Global flags work at the app, checksum, and verify levels.
  • --report is independent of stdout. It still writes object records and the final summary as JSON Lines when explicit quiet suppresses stdout.
  • Output transport does not change --fail-on decisions.

The distinction matters because MCLI’s historical globalQuiet has two inputs: an explicit quiet flag and an automatic non-TTY state used to disable progress UI. Changing that global would risk re-enabling progress output across copy, get, put, mirror, and other commands.

The selected repair is command-local. It walks the full CLI context chain for explicit quiet/JSON flags because the CLI library’s GlobalBool stops at the nearest ancestor flag set. It also restores JSON Lines mode inside the checksum action because nested Before hooks can reset it after an app-level --json. No other command’s progress or output behavior changes.

Report, secrets, and operational cost

Report files are created with mode 0600, must not already exist, and contain metadata/results rather than object bodies or SSE-C keys. The manifest likewise contains only bucket, key, and optional VersionID.

Verification downloads every supported object body. Operators should use --dry-run, --max-size, time filters, --max-workers, and the global download limit to bound cost and load. NO_CHECKSUM and UNKNOWN_* counts must remain visible; neither may be presented as successful verification.

What a mismatch proves

A mismatch proves only that the additional checksum returned at verification time does not describe the logical bytes returned at verification time. It does not prove that a particular historical compression defect created the object, and it is not an external source-of-truth comparison.

Do not overwrite checksum metadata in place. Audit and classify first. For a confirmed, operationally relevant mismatch, prefer a new key or new version, verify the replacement, then switch consumers deliberately. Leave UNKNOWN_* objects out of automatic repair.

Verification record and release boundary

The local acceptance matrix covers TTY human/JSON, non-TTY pipes, regular-file redirects, app/parent/leaf JSON and quiet flags, environment quiet, report under quiet, report-write failure, and MISMATCH/UNKNOWN exit status. It also includes real historical MATCH, MISMATCH, and unsupported-composite objects on a local S3 server.

The command shipped in the final mcli 20260903 release from a signed tag at the tip of main, with the functional suite - including a checksum verification run against a real SILO server - green for that commit, and pgsty/mc#5 is closed. Server 20260903 images bundle that client. A production audit remains a separate execution with its own evidence.

JSON consumers should check schemaVersion: 1 and distinguish per-object records (type: object) from the final summary (type: summary). --max-workers defaults to 4 and accepts 1–64; it bounds concurrent object work, not total memory or server I/O.

4.25 - Optional Checksums, Mandatory Failure: Repairing UploadPart and UploadPartCopy Compatibility

Release check (2026-09-16): the original repair described here is included in Server 20260903. Dated review and test accounts below record their original evidence, not a still-pending release or acceptance of a particular production installation. Later source changes and component selections are in the version matrix.

This is the complete design and implementation record for SILO #46. The repair was not merely a changed if statement. One apparently optional S3 header reached into multipart completion semantics, copy responses, compression and encryption pipelines, compatibility baselines, and release verification.

Status: merged into main as 7fea6d5a5 on 2026-08-24 (pgsty/silo#46 closed); included in Server 20260903; deployment verification is installation-specific.
Owner: pgsty/silo, the SILO server repository.
Tracking: #46.
Independent follow-ups: #63 CopyObject + compression checksum, #64 federated UploadPartCopy checksum.
Adversarial review: local Claude Code, Fable 5, --effort max; final verdict GO, with no blocking findings.

Too Long; Didn’t Read (TL;DR)

A multipart upload splits a large file into smaller parts. A client may attach a checksum to each part so the server can verify the transfer, but AWS defines that checksum as optional. SILO used to treat it as mandatory: an ordinary UploadPart failed without one, and UploadPartCopy could never work because it has no part-body checksum to provide.

After the repair, SILO still validates a checksum when the client sends one. When the client omits it, SILO computes the checksum while reading the original bytes and saves the result. This happens before compression and encryption, requires no second read, and changes no on-disk format. The result is AWS-compatible behavior without weakening data integrity.

Decision

When a multipart upload declares a checksum algorithm in CreateMultipartUpload, SILO applies this contract:

  1. If the client supplies a part checksum, the server continues to validate it. A wrong value or algorithm fails and is never hidden by fallback computation.
  2. If the client omits the part checksum, the server computes it in one pass with the MPU algorithm over the logical plaintext stream, before compression and encryption, and persists the result.
  3. A normal UploadPart echoes a checksum response header only when the client supplied the checksum. A server-computed fallback is not echoed.
  4. UploadPartCopy has no client part-body checksum, so the server computes the value and returns it in CopyPartResult.
  5. ListParts returns the persisted part checksum.
  6. FULL_OBJECT completion continues to linearize the full checksum from stored part checksums. COMPOSITE completion continues to require a checksum for every part; clients can recover those values with ListParts.
  7. Computation occurs during the existing read. Completion never re-reads the entire object merely to manufacture missing state.

In one sentence:

The optional input is the client-provided checksum value, not the server’s responsibility to maintain a consistent checksum-enabled MPU.

How we found it

The defect surfaced while investigating a different multipart checksum issue, #31.

#31 concerned CompleteMultipartUpload: for FULL_OBJECT, a client can complete with part numbers, ETags, and an optional full-object checksum without retaining every part checksum in the completion XML. Tracing that path backward exposed a stronger, earlier condition in erasureObjects.PutObjectPart:

if cs := fi.Metadata[hash.MinIOMultipartChecksum]; cs != "" {
    if r.ContentCRCType().String() != cs {
        return InvalidArgument{/* checksum missing */}
    }
}

Once an MPU declared a checksum algorithm, every UploadPart had to carry the matching x-amz-checksum-* value. Omitting it returned:

400 InvalidArgument:
checksum missing, want "CRC32", got ""

API-level probes reproduced the behavior on both the single-drive and erasure backends.

Reviewing CopyObjectPartHandler raised the severity from a client-configuration incompatibility to P0. UploadPartCopy has no request body for the caller to checksum. The handler reads the source object, constructs an internal reader, and eventually enters the same PutObjectPart implementation. There is no client header and no SDK setting that can repair the request. Every checksum-enabled MPU therefore rejected UploadPartCopy by construction.

What AWS requires

This cannot be decided by saying that MinIO has historically behaved a certain way. The S3 protocol is the authority.

The AWS UploadPart API describes each algorithm-specific checksum header as something that “can be used as a data integrity check.” More importantly, its response fields say that the checksum is present only when it was provided in the request.

The AWS UploadPartCopy API is different: when the MPU was created with an algorithm, the copy result contains that part checksum. There is no copy request body, so this is necessarily a server-computed value.

The AWS ListParts API is the standard way to recover checksums for parts in an upload that is still in progress.

The algorithm/type matrix also rules out treating the repair as one Boolean flag:

Algorithm FULL_OBJECT COMPOSITE
CRC64NVME Supported Unsupported
CRC32 / CRC32C Supported Supported
SHA1 / SHA256 Unsupported Supported

FULL_OBJECT is limited to CRCs that can be linearized, but SHA1 and SHA256 still need correct per-part digests for COMPOSITE completion.

SDK configuration makes the gap practical. Current AWS SDKs usually calculate request checksums when an operation supports them, but users can choose request_checksum_calculation = when_required, and low-level callers can initiate an algorithm without repeating it on every part. S3 accepts those requests; SILO did not.

Why removing the check is not a fix

The most tempting patch is to delete the comparison and allow a checksum-less part to proceed. That only moves the failure to completion.

SILO does not reconstruct and re-read all object bytes during MPU completion. It reads ObjectPartInfo.Checksums from each part.N.meta:

  • a missing entry immediately becomes InvalidPart;
  • FULL_OBJECT calls Checksum.AddPart, combining digests with their part lengths;
  • COMPOSITE concatenates the raw digest bytes and hashes them into the object checksum.

The actual invariant is therefore:

checksum-enabled MPU
        => every committed part has a checksum for the MPU algorithm

Deleting the upload check without filling the metadata would make UploadPart appear successful, leave ListParts incomplete, omit the UploadPartCopy response value, and fail later during completion. A delayed failure is harder to diagnose than the original immediate one.

Alternatives considered

Option Benefit Fatal problem Decision
Delete the strict check Smallest diff Part metadata still lacks the checksum; completion must fail Rejected
Relax only FULL_OBJECT Unblocks some default CRC clients Leaves COMPOSITE and SHA incompatible; cannot close #46 Rejected
Re-read every part at completion Avoids storing a digest during upload Adds O(object size) second-pass I/O and still cannot fix ListParts or the copy response Rejected
Always return the server value from normal UploadPart Makes federation forwarding easy Violates the AWS response contract Rejected
Copy the AIStor implementation exactly Commercial precedent CRC-only fallback and a transformed-stream placement risk Rejected
Compute and persist in one pass over logical plaintext Complete protocol behavior, no second I/O, CRC and SHA support Requires an explicit plaintext checksum reader distinct from the storage reader Accepted

What the commercial edition taught us

We downloaded and verified the then-current MinIO AIStor RELEASE.2026-08-07T18-34-35Z. Without a commercial license the server enters offline mode and denies S3 operations, so the evidence came from Go pclntab and ARM64 disassembly, not a black-box compatibility run.

The static analysis showed that AIStor already:

  • installs a server hasher when the client checksum is absent;
  • persists the result in part metadata;
  • exposes checksum fields in CopyPartResult.

It nevertheless applies fallback only to CanMerge() algorithms—CRC32, CRC32C, and CRC64NVME. SHA1/SHA256 COMPOSITE still follows the old checksum missing path. More importantly, the hasher is attached in the object layer to the current r.Reader; under compression or encryption that reader may already represent transformed storage bytes.

AIStor validated the general direction—compute and store—but not an implementation that SILO could copy mechanically.

How adversarial review overturned the first design

The first plan tried to centralize every decision inside erasureObjects.PutObjectPart: read the MPU metadata in the object layer and install a server hasher when the incoming reader had no client checksum. It looked attractive because all internal callers would share one rule.

The first Fable 5 Max adversarial review found that this design was wrong for compression.

newS2CompressReader is not a lazy wrapper. Construction immediately launches a goroutine:

go func() {
    _, err := io.Copy(comp, r)
    // ...
}()

The S2 writer also reads several blocks concurrently. After constructing the compressor, the handler still performs option parsing, encryption preparation, and the object-layer call. By the time PutObjectPart installed a hasher, the plaintext reader could already have lost several MiB:

  • a large part would get a checksum with a missing prefix;
  • a small part could reach EOF before installation and produce no result;
  • mutating ServerSideHasher concurrently with Read would be a data race.

That finding changed the responsibility split:

The handler installs the hasher before any eager transform starts; the object layer validates the algorithm, requires a result, and persists it atomically.

This was the decisive turn in the design. Putting logic in the lowest layer may look more uniform, but stream correctness depends equally on when bytes begin moving and which representation of those bytes a layer can see.

Final implementation

A dedicated logical checksum reader

PutObjReader originally distinguished two concepts:

  • Reader, the stream sent to storage, possibly compressed or encrypted;
  • rawReader, used by older ETag and checksum code.

Under compression, even rawReader may not directly see plaintext; it can merely carry an ETag through an etag.Tagger chain. The repair therefore did not overload it. It added an unexported field:

checksumReader *hash.Reader

This reader always represents the logical S3 part bytes. WithEncryption can replace the storage Reader, but it must preserve checksumReader.

Unexported accessors on PutObjReader then:

  • return the effective client or server checksum type;
  • prefer the client value whenever it exists;
  • otherwise return the server result finalized at EOF.

Keeping the mechanism unexported minimizes public Go API growth and gives #63 a shared internal path without prematurely changing ordinary CopyObject behavior.

Preparing the hasher before transformations

prepareMultipartChecksumReader loads the algorithm and checksum type saved with the MPU:

  1. no declared algorithm means no work;
  2. an existing client checksum is compared by base algorithm;
  3. a wrong algorithm preserves the InvalidArgument rejection;
  4. an omitted client checksum installs the corresponding server hasher on the plaintext reader.

For normal UploadPart:

  • the compressed path prepares actualReader after request-checksum parsing but before newS2CompressReader;
  • the uncompressed path prepares the request hash reader before the encryption reader is constructed.

For UploadPartCopy:

  • a checksum-enabled MPU first gets an inner hash reader over the logical source range;
  • a range copy hashes only the selected bytes;
  • compression and destination encryption start only after that reader is ready.

The object layer remains authoritative

Early handler preparation does not replace the storage invariant. erasureObjects.PutObjectPart still:

  • re-parses the expected MPU algorithm;
  • requires an effective checksum type that matches;
  • obtains the checksum map after erasure encoding finishes;
  • reports an internal error instead of committing if an enabled algorithm has no result;
  • writes the checksum with the ETag, sizes, and index into part.N.meta, then atomically renames the part.

An internal caller that bypasses the HTTP handler without preparing a valid checksum is therefore rejected just as before. It cannot silently commit a part that violates the MPU invariant.

CopyPart response shape

CopyObjectPartResponse gained the five algorithms supported by this source tree:

ChecksumCRC32
ChecksumCRC32C
ChecksumCRC64NVME
ChecksumSHA1
ChecksumSHA256

All are omitempty, so an MPU without checksums produces the old XML. Normal UploadPart still uses the existing TransferChecksumHeader and echoes only a client request value; fallback computation does not alter that response.

Why it works

After the repair, the data flow is:

logical plaintext part
        |
        +--> client checksum verifier (if supplied)
        |         or
        +--> server-side hasher (if omitted)
        |
        v
compression (optional)
        |
        v
encryption (optional)
        |
        v
erasure encode / storage
        |
        v
persist ETag + size + logical part checksum atomically

This satisfies four requirements that previously appeared to conflict:

  1. Protocol compatibility: omitting an optional header succeeds.
  2. No integrity downgrade: a supplied client value is still checked end to end and is never hidden by server fallback.
  3. Correct object semantics: the checksum covers logical S3 bytes, not compressed data or ciphertext.
  4. Controlled cost: hashing shares the existing read and adds CPU, not a second disk or network pass.

EOF has a precise role. hash.Reader finalizes ServerSideChecksumResult only when it reaches EOF. Closing the compression pipe synchronizes the compressor goroutine with the storage read; the object layer reads the result only after encoding returns. Targeted -race tests verified that concurrency boundary.

The compatibility-baseline blocker

The five new CopyObjectPartResponse fields are exported Go API. SILO’s buildscripts/rebrand-guard rescans imports, environment variables, headers, routes, storage markers, and exported symbols, then compares them in both directions with buildscripts/rebrand-guard/compat-baseline.json. An unacknowledged symbol makes CI fail.

After recording the five #46 fields, the guard still reported two additions:

internal/config/notify:notify:type:LegacyDatabaseTargetError
internal/config/notify:notify:method:LegacyDatabaseTargetError.Error

They did not come from #46. They belong to the earlier database-notification repair f1ba68358 on the local main branch. The cmd startup path intentionally needs the exported type for errors.As, but that earlier commit had not updated the compatibility baseline. Every later change based on that HEAD would therefore fail the CI guard.

We chose “option A”: acknowledge the two notification symbols as part of their original repair while retaining the five #46 fields. The final baseline diff is exactly seven additions and zero deletions, and the guard reports:

exported=9021
Silo rebrand compatibility baseline is unchanged

This does not disable the check. Exact set equality means that acknowledging a nonexistent symbol also fails. The change explicitly records two intentional compatibility-surface additions.

golangci-lint has not yet run locally; it remains a remote go.yml gate. Green local go test, go vet, race, and rebrand-guard results do not substitute for green remote CI.

Verification evidence

The new tests execute 76 subtests across:

  • CRC32, CRC32C, and CRC64NVME FULL_OBJECT;
  • CRC32, SHA1, and SHA256 COMPOSITE;
  • correct client checksums, wrong algorithms, and wrong values;
  • absence of a server-computed checksum in normal UploadPart responses;
  • server values in UploadPartCopy responses and ListParts;
  • a real 5 MiB + 1 KiB two-part full-object merge;
  • zero-length parts and overwriting the same part number;
  • a range copy whose SHA256 covers only the copied interval;
  • single-drive and 16-drive erasure backends;
  • default, versioned, compressed, encrypted, and compressed-plus-encrypted modes;
  • explicit SSE-C and SSE-S3.

Local validation included:

go test -race ./cmd -run '^TestAPIUploadPartServerSideChecksum' -count=1
go test ./cmd -count=1
go test ./... -count=1
go vet ./cmd
git diff --check
go run ./buildscripts/rebrand-guard

All passed. Two subsequent Claude Code Fable 5 Max implementation reviews and the final acceptance review returned GO with no blocking findings.

Cost, risk, and release boundary

When a client omits its value, the server performs one additional hash over the part. CRC cost is small; SHA costs more CPU. Both share the read that already had to occur, without buffering an entire part in memory or adding a completion-time second pass.

During a rolling upgrade, old and new nodes may answer the same checksum-less request differently: a new node accepts it while an old node returns 400. ObjectPartInfo.Checksums did not change format, so stored data remains downgrade-readable, but client-visible behavior stabilizes only after all serving nodes have upgraded. The release note must call that out.

The original local implementation later merged and shipped in Server 20260903. Its historical local verification record does not establish the state of any production deployment.

Why two follow-ups remain separate

Adversarial review found two related but independent issues.

#63: CopyObject + compression

Ordinary CopyObject can also attach a server-side checksum to a transformed stream. It shares the root cause and the new checksumReader mechanism, but it is a different API with a different test matrix and rollback boundary. We chose a separate repair and require that PR to reuse this plaintext-reader contract instead of inventing a second abstraction.

#64: legacy federation

Legacy etcd federation turns UploadPartCopy into an ordinary remote UploadPart. Under the AWS response semantics preserved here, that remote request does not return a server fallback value, so the proxy may still lack the checksum required for CopyPartResult. A follow-up must independently choose between a remote-returned value and an ETag-verified ListParts fallback. It must not make all external UploadPart responses non-compliant merely to simplify an internal proxy.

Separating them does not abandon consistency. Consistency is maintained through one shared rule:

Every server-computed S3 checksum binds to the logical plaintext stream, is installed before any eager transform, and is validated and persisted by the object layer that owns the storage invariant.

Lessons retained

The repair leaves lessons more durable than its individual lines of code:

  1. An optional header does not make internal state optional. If the protocol lets the client omit a value, the server must produce the state its own completion path needs.
  2. Request acceptance and response disclosure are separate contracts. A normal UploadPart may compute internally and still omit the value; UploadPartCopy must return it.
  3. Stream layers are defined by byte semantics. The lowest layer is not automatically correct if it no longer sees logical bytes, and an eager goroutine turns “install later” into a race.
  4. A commercial implementation is evidence, not the specification. AIStor showed the direction and the boundary that could not be copied.
  5. A compatibility guard is a change-acknowledgment mechanism. compat-baseline.json exists to assign every new compatibility surface, not merely to make CI quiet.
  6. Independent defects should ship independently while sharing invariants. #63 and #64 remain separate, but both must cite and obey the checksum-reader contract established here.

The final result is not a broad relaxation. It is a stricter and more accurate boundary: clients may omit optional information; the server may not omit correctness.

4.26 - BadDigest, InvalidRequest, and the CompleteMultipartUpload Checksum Contract

The repairs for #48 and #50 are included in Server 20260903. This record supersedes the August proposal to leave CRC64NVME canonicalization unchanged.

Why validation belongs at completion

Initiation selects the checksum algorithm and object type; each part records its checksum. Completion must validate the final value and any explicit type assertion against that stored contract. A request cannot change COMPOSITE to FULL_OBJECT just because both share the same base algorithm, or bypass the assertion by omitting the final digest.

The repair distinguishes an omitted type from an explicit type and normalizes only internal representation flags. It uses operation-specific error types, leaving UploadPart and the global checksum-mismatch mapper unchanged.

Current error contract

All failures below return HTTP 400 and do not commit a new completed object.

Request condition S3 error
Wrong full-object or composite object digest BadDigest
Explicit supported type differs from the initiated type, including a type-only assertion BadDigest
Unknown or lowercase type token InvalidArgument
Completion algorithm differs from the initiated algorithm InvalidArgument
Missing required composite part checksum InvalidRequest, naming the algorithm and part
CRC64NVME with COMPOSITE at initiation InvalidArgument
CRC64NVME value with COMPOSITE rejected by checksum parsing at completion InvalidArgument
Bare COMPOSITE type at completion of a CRC64NVME FULL_OBJECT upload BadDigest
Incorrect client checksum during UploadPart Existing XAmzContentChecksumMismatch

For FULL_OBJECT, part checksums may be omitted; any supplied values remain validated. Matching type-only assertions are allowed. An omitted optional type does not assert COMPOSITE. SHA1/SHA256 with FULL_OBJECT are rejected by the algorithm/type parser before the stored-type comparison.

CRC64NVME decision and source evidence

The original review deferred #50 pending evidence; that is historical, not the current contract. PR #93 rejects the invalid algorithm/type combination in headers and trailers, and PR #96 removes completion’s canonicalization. Both precede the 20260903 tag. Completion can fail at either parsing or stored-type comparison, which explains the two distinct error codes above.

The initial mapping is in PR #74, with the explicit type follow-up 7e079ff05 merged through PR #85. Current handler tests exercise full-object and composite mismatches, type-only assertions, invalid tokens and both CRC64NVME rejection stages. This is committed regression coverage, not a claim that this documentation update reran all Server tests.

Separate streaming-checksum follow-up

PR #143, following @cbornet’s #107, handles an aws-chunked request that advertises x-amz-trailer but supplies its checksum in a header, as used by the AWS Java SDK v2. It also rejects an invalid header-delivered value instead of dropping validation. This later repair is on main and not in Server 20260903. It is separate from completion error mapping.

Compatibility and remaining boundaries

Callers that inspect error codes now see checksum/type failures as BadDigest and missing composite values as InvalidRequest. Successful checksum-free uploads and ETag semantics are unchanged. No metadata migration is required.

Several narrower differences remain: providing a final digest when initiation recorded no checksum is rejected as BadDigest; composite part-count and value mismatches share one description; a -N suffix on a full-object checksum is not itself validated as a part count. These are documented observations, not claims of separately filed public issues or complete AWS parity.

4.27 - ListMultipartUploads: Implementation, Upgrade Contract and Design History

This is the problem, design, and decision record for SILO issue #79.

September 16 implementation and upgrade contract

PR #198 retains mr javad seydi’s original metadata-and-scan contribution and adds maintainer fixes for disappearing markers, incomplete discovery, cancellation confirmation and upgrade diagnostics. The pre-release compatibility follow-up restores legacy as the default and makes strict mode an explicit process-environment opt-in. This section describes those source changes. It is not included in Server 20260903, and does not establish production performance or deployment acceptance. The August 30 analysis below remains a historical design record.

Listing and cancellation

New uploads persist their bucket and object key in the existing xl.meta; completion removes these upload-only fields. Strict listing discovers durable uploads across pools and sets, verifies metadata with the existing read quorum, and applies prefix, delimiter, CommonPrefixes and a global page limit of at most 1,000. Restarting a node or choosing another endpoint does not depend on rebuilding its upload cache.

Ordering is (key, initiation time from the native upload ID, encoded upload ID). A returned marker defines this boundary even after the corresponding upload is completed or canceled. This supports clients that echo the server’s markers; it does not promise arbitrary lexical comparison of random upload IDs. There is no snapshot across pages under concurrent mutations. Without key-marker, upload-id-marker is ignored. With a key marker, invalid base64 retains the existing 404 response; a decodable but unsupported native ID returns 400. Unsupported persistent IDs are legacy records, never assigned a fabricated current initiation time.

Directory discovery requires floor(N/2)+1 successful drive scans per set. For example, two available drives out of four are insufficient and return 503, even when metadata could still be read from two copies. A source-drive identity read may exclude another bucket only after validating its bucket/key hash; uncertain identities require a quorum read. Strict mode returns MultipartListingNotReady (503) for legacy records and MultipartListingMetadataInvalid (503) for invalid identities. It never silently switches the whole request to cache-based listing.

In default legacy mode, Abort retains the released read quorum, best-effort cleanup and pool-order return behavior, avoiding new 503 responses for previously successful cancellations. In strict mode it checks every relevant pool and requires floor(N/2)+1 deletion acknowledgements per set. Remnants below read quorum can still be retried. When a majority was already absent but remnants were observed, those observed copies must be cleaned successfully; successful responses from empty drives cannot mask their deletion failures. Unknown pools or insufficient confirmations still return 503.

In both modes, an Abort with the wrong key or bucket cannot evict another valid upload’s cache entry. The S3 HTTP layer retains its existing idempotent response: a nonexistent upload also returns 204; malformed input, authorization and quorum errors remain errors. 204 does not prove that every physical copy has been deleted. Offline part data may still need later cleanup.

Unresolved creation-write boundary: these confirmations do not fence a physical creation write that continues after its caller receives a storage timeout. A fault-injection test reproduces a 16-drive/EC:8 case where seven delayed writes and seven offline old copies restore a writable upload after acknowledged cancellation. The test preserves this known limitation; its passing status is not a repair claim. A durable creation fence needs a separate storage-consistency design.

Coordinated upgrade

The default is legacy, retaining the old exact-key/cache listing limitations. An ordinary upgrade does not require pausing production or draining uploads to enable the new listing. The mode is read only from the server process environment, MINIO_API_MULTIPART_LISTING=legacy|strict; no shared configuration key is added. An unset value selects legacy. An invalid value logs a diagnostic and falls back to legacy while preserving other API settings. Do not set this mode with mcli admin config set.

Only before opting into strict mode must operators upgrade all writers, stop introducing old-format uploads, finish or abort legacy uploads using known keys/IDs, run the read-only preflight below, and verify scan capacity against their workload. Once these checks pass, set MINIO_API_MULTIPART_LISTING=strict in every server’s service environment and restart; check the effective mode at each endpoint. Restart paginated traversals after changing modes. Issue #79 remains open for the default listing limitations; this batch does not claim a complete repair.

If a development build already persisted api multipart_listing, back up configuration and record API values, upgrade all configuration writers to the patched version, prevent concurrent configuration writes, then remove only that historical key:

mcli admin config reset ALIAS api multipart_listing
mcli admin config get ALIAS api

The patched server ignores the historical key’s mode value without automatically rewriting shared configuration or deleting history. The config get/export views omit retired keys, so their absence from those views alone does not prove deletion. Require a successful targeted reset, then verify preserved values after restart and during a controlled rollback check; do not reset the whole api subsystem. A development build can reintroduce the key on its next configuration write, and history containing that key should not be replayed directly. Existing API values can temporarily be pinned through their corresponding environment variables where needed, but that does not replace persistent cleanup. This procedure covers this API configuration change only; other rollback constraints require separate checks.

The read-only, SigV4-authenticated endpoint GET /minio/admin/v3/multipart-preflight requires admin:StorageInfo. For example, with credentials and an endpoint supplied by the operator:

curl --aws-sigv4 'aws:amz:us-east-1:s3' \
  --user "$SILO_ACCESS_KEY:$SILO_SECRET_KEY" \
  "$SILO_ENDPOINT/minio/admin/v3/multipart-preflight"

The report contains mode, ready, complete, scannedEntries, legacyUploads, and per-pool/set drive coverage, uncovered drive indexes and oldest legacy initiation time. It bypasses upload caches, inspects even suspended pools and detects minority legacy copies. ready=true requires every drive to be inspected, no unreadable candidate metadata and no observed legacy copies. It cannot attest that every writer has been upgraded or that no concurrent writer will introduce a legacy record. Offline drives, scan errors, timeouts and budget exhaustion prevent readiness; an incomplete count is not a zero count. Rerun after drives return and before enabling strict mode.

For uploads whose original key/ID has been lost, the existing stale-upload cleanup scans each server’s local drives. Age is measured from creation, not recent part activity. Retain the existing cleanup policy, wait and verify actual drain; the default 24-hour expiry and 6-hour interval do not guarantee drain completion. Do not shorten expiry to accelerate an ordinary upgrade, since this can also remove active long-running uploads. Continue using default legacy mode if drain cannot be established. This batch changes neither cleanup policy nor the available deletion APIs.

The shared maxUploadsList cap changes from 10,000 to 1,000 in both modes; clients that assumed one response contained everything must paginate.

A legacy upload lacks bucket/key identity and cannot safely be excluded as belonging to another bucket, so it can block strict listing for any bucket. A valid new-format identity belonging to another bucket can be filtered earlier. This does not impose deployment-wide 503 responses on default legacy listing.

Scan capacity and evidence limits

Each process admits two scans, with 16 identity workers and four full-metadata workers per scan. Directory reads pass a finite count with overflow detection. The aggregate budget is 100,000 returned directory entries, including repeated entries on different drives and hash directories; it is not a promise to list 100,000 unique uploads. Concurrent directory calls may already be in flight when the aggregate budget is exceeded. Overflow returns SlowDown (503), never a successful partial page. A 30-second context budget stops further scheduling; admission stays held until scan workers exit. This is neither a precise memory ceiling nor a guarantee that a canceled physical system call stops immediately.

As a budget illustration only, one upload per unique key on every drive of an N-drive set costs about 2 × N entries across the two directory levels: 100000 / (2 × N) uploads, or about 3,125 at N=16. Multiple uploads under one key share the hash-directory cost; other sets/pools, stale directories and timeouts also matter. This is not a universal upload-count limit.

Every page still rescans durable state: total enumeration cost grows with both stored candidates and page count. The tests cover missing markers, multi-pool coverage, partial-deletion retries, identity fallback, RPC directory bounds, cancellation admission and the known late-write counterexample. Temporary multi-node and maintained-client checks establish functional behavior for their recorded environment. Production-scale latency and foreground-load impact remain deployment-specific acceptance work; merging the source does not certify them.

A September 16 temporary Docker Desktop arm64 run used two nodes, four APFS-backed bind volumes and 11,000 uploads across two buckets, while other local validation was running. One 1,000-entry page for the 10,000-upload target bucket took 18.7 seconds; two concurrent requests returned retryable SlowDownRead responses after about 25–27 seconds. These observations do not meet the provisional five-second page target and are not an isolated SSD benchmark. Capacity and foreground-load acceptance remain open; the bounded scanner must not be advertised as a large-scale performance fix.

Historical analysis (2026-08-30): “current”, “recommended” and gate language below describes that review and proposal. The September 16 section above defines the implementation and remaining limits. Capability advertisement and the old one-day drain estimate are not current guarantees.

The problem in plain language

Imagine that four large files are still being uploaded:

tables/a/part-1
tables/a/part-2
tables/b/part-1
other/file

An S3 client asks, “show me every unfinished upload below tables/.” AWS S3 returns the first three. SILO currently treats tables/ as if it were the complete name of one object, looks for exactly that object, and returns an empty list.

If the client removes the prefix and asks for every unfinished upload in the bucket, SILO takes a different shortcut: it reads a process-local memory cache. That cache may contain all four uploads on the node that created them, but it does not survive a restart and is not authoritative across nodes. The upload data is still on disk; the list is wrong.

This is why the defect is more serious than one ignored query parameter. Cleanup tools can receive 200 OK, conclude that no unfinished uploads exist, and report success while uploads remain on disk. The server is not losing committed objects, but it is giving callers a false view of unfinished work.

Executive decision

SILO should fix this behavior if it intends to keep advertising practical S3 compatibility.

The repair is justified because the current endpoint silently claims success, behaves differently after restart or node switching, and breaks standard prefix-based cleanup and pagination. The default 24-hour stale-upload collector limits storage accumulation on default configurations, but it does not make the API result truthful.

The repair is not a small change. Existing upload directories contain only a one-way hash of the bucket and object key, and the original key is not stored in their xl.meta. A correct implementation must begin persisting that identity for new uploads, discover candidates with erasure-aware quorum rules, apply S3 semantics globally across pools and sets, and handle legacy uploads during a rolling upgrade.

The recommended direction is therefore:

  1. record the bucket and object key in the upload’s existing quorum-written metadata;
  2. build a bounded on-demand scan as the durable correctness path;
  3. keep any cache only as a rebuildable optimization;
  4. enable strict S3 behavior only after every writer has upgraded and all keyless legacy uploads have drained;
  5. consider a durable secondary index only if measurements prove that scanning cannot meet a product-approved service-level objective.

Sources and provenance

The problem statement and proposed design are grounded in five kinds of evidence.

The S3 contract

The AWS ListMultipartUploads API defines the public contract for general-purpose buckets:

  • prefix selects every upload whose key starts with that string;
  • delimiter groups matching keys into CommonPrefixes;
  • max-uploads limits a page, with 1,000 as the documented maximum;
  • key-marker and upload-id-marker continue a truncated listing;
  • upload-id-marker is ignored when key-marker is absent;
  • results are ordered by object key, then by initiation time for uploads with the same key.

AWS documentation does not settle every implementation edge unambiguously. Equal timestamps, invalid or out-of-range max-uploads, URL encoding, marker boundaries, and the way CommonPrefixes consume a page should be captured once against AWS and stored as fixtures before implementation.

The reported defect

Issue #79 supplied a self-contained signed reproducer against pgsty/silo:latest and compared SILO with AWS, RustFS, SeaweedFS, and Garage. Its four central observations reproduce:

Request Required behavior Observed SILO behavior
prefix=t/ return the three keys beginning with t/ returns no uploads
max-uploads=1 return one item and continuation markers returns every cached upload
key-marker=t/a_b/p2 continue after that key returns every cached upload
prefix=t/&delimiter=/ return grouped CommonPrefixes returns neither uploads nor prefixes

The issue correctly identifies a compatibility failure, but its statement that max-uploads is always ignored and IsTruncated is always false is broader than the implementation. Those claims hold on the empty-prefix cache path used by the reproducer; the exact-object path can honor max-uploads and upload-id-marker and can set IsTruncated.

The upstream design history

The behavior was inherited rather than invented by SILO:

  • MinIO PR #5248 deliberately removed prefix-based listing from the erasure backend in 2017 “to simplify” multipart support.
  • MinIO PR #20407 added the empty-prefix multipart cache in 2024, mainly for Alluxio tests.
  • A 2025 report of the same exact-key behavior, MinIO issue #20989, was closed as working as intended.
  • SILO’s current S3 compatibility reference already records the exact-object-name divergence, although it did not explain the cache, pagination, marker, delimiter, or restart limitations before this design record.

This history explains why the code looks deliberate. It does not make the endpoint compatible with the AWS contract.

Source review

The current source has two mutually exclusive listing paths:

erasureServerPools.ListMultipartUploads
  prefix == ""  -> return entries from node-local mpCache
  prefix != ""  -> hash prefix as a complete object key
                    -> select one set
                    -> list one sha256(bucket/object) directory

The important locations are:

  • cmd/erasure-server-pool.go: empty-prefix mpCache, per-pool concatenation, and the internal exact-object lookup used by NewMultipartUpload;
  • cmd/erasure-multipart.go: exact-object listing, upload directory construction, stale-upload cleanup, and the quorum write for a new upload’s xl.meta;
  • cmd/erasure-sets.go: hashing a supplied object name to one erasure set;
  • cmd/bucket-handlers.go: public request validation, including a 501 NotImplemented guard when key-marker does not share the request prefix;
  • cmd/object-api-multipart_test.go: a large expected-results table whose final assertion block checks only echoed scalar fields, not the returned uploads, prefixes, markers, or truncation state.

Independent reproduction and adversarial review

The issue scenario was independently reproduced against the reviewed SILO source with a single-node server and SigV4 requests. Additional probes established that:

  • an exact object key can paginate its own uploads;
  • upload-id-marker currently affects that exact-key path even without key-marker, contrary to AWS;
  • NextKeyMarker remains empty on an exact-key truncated page;
  • max-uploads=0 behaves as unlimited in the current path;
  • a plain server restart empties the bucket-wide view while exact-key lookup still finds the on-disk uploads.

A second adversarial architecture review challenged the storage, quorum, migration, suspended-pool, mixed-version, and performance assumptions. The corrections from that review are incorporated below; this record does not treat an AI review as a substitute for code tests or an AWS conformance capture.

What the code actually does

Empty prefix: a volatile node-local view

With no prefix, the pool layer returns every MultipartInfo for the bucket from mpCache, sorted only by initiation time. It does not apply max-uploads, key-marker, upload-id-marker, or delimiter, and it does not compute continuation markers or IsTruncated.

The cache is initialized empty at process startup. Creation populates only the node that handles the request. Completion and abort delete cache entries, including peer notifications in some paths, but creation has no equivalent durable cluster-wide population or startup rebuild. Consequently:

  • a restart can change a non-empty listing into an empty one;
  • two nodes can return different answers for the same bucket;
  • a successful response is not evidence that the server has enumerated durable upload state.

Non-empty prefix: an exact object lookup

With a non-empty prefix, the string is passed through object hashing as though it were a complete object name. One erasure set is selected, and listing reads the directory derived from sha256(bucket/object).

This path can enumerate multiple upload IDs for that exact object. It sorts them by initiation time, applies its upload-id-marker, stops at max-uploads, and sets IsTruncated. It still does not implement lexical prefix matching, CommonPrefixes, general key-marker semantics, or NextKeyMarker.

Multiple pools make pagination less correct

For a non-empty request on a multi-pool deployment, the pool layer invokes each active pool with the same maximum and concatenates the results. It does not perform a global ordered merge or recompute page boundaries and next markers. A request for N items can therefore collect up to N from each pool.

Suspended pools are skipped by listing and by the other public multipart verbs. In-progress uploads left on a suspended or decommissioning pool are therefore inaccessible, not merely unlisted. That is a related lifecycle defect, but listing alone must not advertise handles that PutObjectPart, ListParts, CompleteMultipartUpload, and AbortMultipartUpload cannot use. Pool drain or forced abort should be designed as a separate cross-verb change.

Why current uploads cannot be backfilled

The multipart namespace is flat:

.minio.sys/multipart/<sha256(bucket/object)>/<upload-id>/xl.meta

The hash is one-way. The original bucket and key are not encoded in the path. They are also not stored as a name field in the current multipart xl.meta; the supplied object name only influences the erasure distribution during newFileInfo construction.

Therefore an all-directory scan can discover that an upload exists, but it cannot determine which bucket or key it belongs to. The current node-local cache cannot repair this reliably because it is incomplete across nodes and disappears on restart.

This rules out a tempting “small” fix: scanning every existing xl.meta and applying prefix filters. New identity metadata or a durable index is required, and old keyless uploads need an explicit migration policy.

Complexity assessment

The semantic algorithm is not the hardest part. The hard part is obtaining a complete, quorum-valid, globally ordered input set without turning a listing call into an uncontrolled cluster-wide metadata storm.

Area Complexity Why
Pure S3 filtering and pagination Medium Rules are finite, but marker and delimiter edge cases need captured AWS evidence.
Persisting bucket/key in new upload metadata Medium It reuses an existing quorum write, but completion, rollback, healing, and replication compatibility must be tested.
Candidate discovery High The namespace mixes every bucket and duplicates each upload across erasure drives. One-disk discovery can miss quorum-valid uploads.
Quorum and concurrent deletion High A scan must reject minority ghosts while tolerating abort, completion, GC rename-to-trash, and transient ENOENT.
Multi-pool global pagination High Results must be merged, sorted, truncated, and marked once across all accessible pools and sets.
Rolling migration High Old writers keep creating keyless uploads; old completers may preserve unknown internal metadata.
Performance and resource control High A bucket request may require inspecting every active upload in the cluster, not only that bucket.

Overall, this is a high-complexity compatibility project with medium wire-compatibility risk and high implementation-correctness risk. It is not a destructive object-format migration: the recommended design adds internal metadata for new incomplete uploads and leaves the existing directory scheme in place.

Compatibility and operational impact

Wire behavior changes

A correct implementation deliberately changes observable results:

  • prefix=foo will match foo, foobar, and foo/..., not only the exact key foo;
  • bucket-wide results will be ordered by key and initiation time rather than only initiation time;
  • max-uploads will actually limit a page;
  • the default and maximum will move from SILO’s current 10,000 constant toward the AWS limit of 1,000, subject to the captured edge-case contract;
  • clients must follow NextKeyMarker and NextUploadIdMarker instead of assuming one response contains everything;
  • delimiter requests will return CommonPrefixes;
  • the current handler-side 501 for a marker outside the prefix will be replaced by the captured AWS semantics.

These are compatibility fixes, but they can break software that accidentally depends on SILO’s old non-S3 behavior. In particular, a client that ignores pagination may see fewer entries after the repair. Strict behavior should therefore be introduced through an explicit release and rollout contract, not silently slipped into an unrelated patch.

Storage-format compatibility

The recommended write path adds the bucket and object key as reserved internal metadata inside the new upload’s existing quorum-written xl.meta. It does not rename multipart directories or create a second transactional write.

Before CompleteMultipartUpload renames upload metadata into the completed object, the new upload-only fields must be removed alongside the multipart checksum fields that are already stripped there.

An old binary completing an upload created by a new binary will not know to remove the new internal keys. They would remain inert and hidden from S3 user metadata, but persist in the completed object’s internal metadata. Rolling-upgrade tests must prove that unknown reserved keys do not disturb healing, replication, metadata comparison, or downgrade reads. The product must then choose between tolerating that residue and adding a scrubber; it must not assume the keys disappear.

Operational cost

Because all buckets share one flat hash namespace, an on-demand scan is O(all active multipart uploads in the cluster), not O(uploads in the requested bucket). Bounded parallelism, cancellation, memory limits, and failure behavior are part of correctness, not optional tuning.

Default SILO configuration expires stale multipart uploads after 24 hours and runs cleanup every 6 hours. Once the last old writer has been upgraded, the keyless population should normally drain within roughly 30 hours. Operators with a larger custom expiry have a longer migration window. A zero value is mapped back to the 24-hour default in the current code; no supported “disabled” expiry value was identified in this review.

The collector bounds default storage accumulation, but does not repair a false listing response. It also does not remove the need to test sustained legitimate multipart activity, failure modes, and custom expiry settings.

Severity

The recommended classification is P1 / high compatibility, not P0:

  • no committed object data loss was demonstrated;
  • no security boundary is bypassed;
  • unfinished uploads remain on disk until completed, aborted, or collected;
  • default stale-upload cleanup bounds accumulation in the ordinary configuration.

It remains high rather than medium because the server returns fabricated success, the answer changes after restart or node switching, and cleanup or quiescence tooling can be misled into false confidence.

Options considered

Option 0: leave the current behavior unchanged

This has no engineering cost and preserves every accidental behavior. It also preserves false 200 OK responses, node-local inconsistency, restart volatility, broken prefix cleanup, and an inaccurate impression of S3 support.

This option is acceptable only if SILO deliberately downgrades the public compatibility claim and treats the endpoint as unsupported. Even then, silently returning an incomplete success is inferior to explicit rejection.

Decision: reject as a long-term position.

Option 1: explicit documented divergence

Reject combinations that SILO cannot honor with a stable NotImplemented-class error and document the exact supported subset. This is operationally honest and much smaller than full compatibility.

It is still a breaking change: tools that currently receive an empty or unbounded 200 OK may begin failing jobs. It also does not produce an S3-compatible endpoint. The error behavior and default release policy must be deliberate.

Decision: acceptable short-term containment if full compatibility is declined or deferred; not a compatibility fix.

Option 2: persist identity, scan durable state, optionally cache

For each new upload, store the bucket and key in reserved internal metadata in the upload’s existing xl.meta. For listing, discover upload directories across accessible pools and sets, validate candidates with erasure read quorum, then run one global S3 semantic layer. A cache may accelerate this path only if it can be rebuilt and reconciled from durable state.

This avoids a second write transaction and keeps the directory layout stable. Its principal cost is the cluster-wide scan.

Decision: recommended, subject to a performance and failure-mode spike.

Option 3: durable bucket-scoped ordered index

Maintain a secondary index ordered by bucket, key, and upload identity. Listing becomes scalable and naturally paginable, but create, complete, abort, healing, rollback, and reconciliation must keep two locations consistent across failures. The design resembles multipart index structures that upstream MinIO deliberately removed while simplifying this subsystem.

Decision: no-go unless measurements show that Option 2 cannot meet the product-approved service-level objective.

Rejected variant: repair only mpCache

Filtering, sorting, paginating, broadcasting creates, or rebuilding the current cache would improve symptoms but would not by itself establish a durable quorum-valid source of truth. A cache-only patch risks producing a more convincing but still incorrect answer.

Decision: reject. A cache can optimize a correct read path, never define it.

1. Freeze the public contract first

Create a recorded AWS fixture suite for general-purpose buckets covering:

  • ordering across keys and multiple uploads of one key;
  • equal initiation times and a deterministic total-order tie-break;
  • prefix and exact-key overlap;
  • key-marker with and without upload-id-marker;
  • upload-id-marker without key-marker;
  • delimiter, CommonPrefixes, and page accounting;
  • max-uploads omitted, 0, 1, 1,000, and greater than 1,000;
  • encoding-type=url;
  • empty pages, final pages, and next-marker values.

The captured responses should become repository fixtures. CI should not depend on live AWS access.

2. Persist recoverable identity in the existing write

At NewMultipartUpload, add reserved internal metadata for the canonical bucket and object key before the existing writeAllMetadata quorum write. The exact key names are an implementation detail, but they must be versioned, unambiguous, size-bounded by the existing object-key limits, and excluded from client-visible metadata.

At successful completion, delete those upload-only keys before copying fi.Metadata into the final object metadata and before renameData. Abort and stale cleanup already delete the entire upload directory and need no separate index operation.

3. Separate discovery from validation

Candidate discovery and candidate validity are different questions.

For every accessible, non-suspended pool and set:

  1. list candidate hash and upload directories from all online drives required by the configured list-quorum policy;
  2. union and deduplicate those names;
  3. read the candidate xl.meta through the normal erasure metadata machinery;
  4. include the upload only when its metadata is quorum-valid and contains a valid bucket/key identity;
  5. tolerate a candidate disappearing during abort, completion, or stale cleanup;
  6. under strict list quorum, fail the request rather than return a partial 200 OK when a required set cannot be evaluated.

Using the first healthy disk for discovery is insufficient: that disk may have been offline when a still-quorum-valid upload was created.

4. Apply semantics once, globally

Feed the validated candidates from all pools and sets into a pure semantic layer. The layer owns bucket filtering, prefix, delimiter grouping, ordering, markers, maximum-page accounting, URL encoding, IsTruncated, and next markers.

Pool-local limits and markers must not be applied before the global merge. The result should be deterministic under duplicate discovery and independent of which node handles the request.

5. Preserve the internal exact-object operation

erasureServerPools.NewMultipartUpload currently calls ListMultipartUploads(bucket, object, ...) to keep another upload for the same object in the same pool. If the public function starts treating that argument as a lexical prefix, foo could match foobar and select the wrong pool.

Introduce a narrowly named internal helper such as FindMultipartUploadPool or ListMultipartUploadsExact. It should use the existing object hash path and must not share the public prefix semantics.

6. Treat cache as an optimization

The existing mpCache may be removed. If retained, it must satisfy all of the following:

  • durable state remains authoritative;
  • startup can rebuild it;
  • create, complete, and abort updates are propagated consistently;
  • reconciliation detects missed events and stale entries;
  • a cold or divergent cache falls back to the quorum-valid scan;
  • correctness tests pass with the cache disabled.

7. Gate strict behavior through rolling migration

Legacy upload records lack bucket/key identity and cannot be reconstructed reliably. Use two externally meaningful modes:

  • legacy mode, the initial upgrade default: new writers persist identity; keyless uploads are counted and drained; the documented response policy for a mixed keyed/keyless population must be selected explicitly;
  • strict mode: activation requires every writer node to advertise the new metadata capability and the observed keyless count to be zero. Discovering a keyless upload afterward is an error with anomaly telemetry, not a silent omission.

A short shadow comparison can help validate the new scanner, but a permanent third operating mode is unnecessary unless the spike finds a need. With default expiry, the expected legacy drain is about one day plus one cleanup interval after the last old writer stops.

There is one unresolved product choice in legacy mode:

Policy Advantage Cost
return the complete keyed subset with documented telemetry keeps tools operating during the bounded drain still returns an incomplete 200 OK that ordinary clients cannot see is incomplete
fail listing while any keyless upload exists never fabricates completeness can block cleanup and existing jobs throughout the drain window

This choice belongs in the ADR. Strict mode has no such ambiguity: it must fail loud if its precondition is violated.

8. Keep suspended-pool lifecycle separate

Listing should initially mirror the accessibility contract of the other multipart verbs and scan non-suspended pools. Adding suspended-pool entries to listing alone would expose uploads that cannot be extended, completed, or aborted.

Open a separate lifecycle design for in-progress uploads when a pool drains: either keep all multipart verbs available until the uploads finish, migrate them, or force-abort them under a documented policy. Do not hide that problem inside #79.

Performance spike and decision rule

Option 2 is preferred because it has one durable write location, but its scan cost must be measured rather than assumed.

Generate 1,000, 10,000, and 100,000 active uploads across a matrix of pools, sets, and drive counts. Measure:

  • cold and warm p50/p95/p99 latency;
  • total and per-drive ListDir operations;
  • metadata-read and internode RPC counts;
  • peak memory and allocation volume;
  • cancellation latency;
  • behavior with slow, offline, healing, and intermittently disappearing drives;
  • simultaneous create, complete, abort, and stale cleanup;
  • first-page and deep-page cost with selective and empty prefixes.

The acceptance threshold is a product decision and must be recorded before interpreting the result. A guessed one- or two-second target is not evidence. If the scan meets the approved target with bounded resource use, reject Option 3. If it does not, use the measurements to design the smallest durable index that solves the demonstrated bottleneck.

Test and release gates

Semantic and unit tests

  • pure table tests generated from recorded AWS fixtures;
  • ordering, marker, delimiter, encoding, truncation, and maximum-edge coverage;
  • property tests ensuring pagination returns each logical upload exactly once;
  • deterministic behavior with duplicate candidates and equal timestamps.

Object and handler tests

  • strengthen the existing object-layer table to assert uploads, common prefixes, markers, and truncation;
  • parse and validate handler XML bodies instead of checking only status codes;
  • verify default and invalid max-uploads handling;
  • test exact-helper pool selection independently of public prefix semantics.

Distributed and failure tests

  • restart equivalence and node-switch equivalence;
  • multiple sets and pools with a single global page boundary;
  • candidate missing from one drive but present at quorum;
  • minority ghost after partial abort;
  • concurrent completion and GC rename-to-trash;
  • unavailable set under every supported list_quorum policy;
  • rolling upgrade, old-writer reintroduction, downgrade completion, and strict-mode gating;
  • unknown internal metadata under healing and replication.

Delivery gates

  1. approve the ADR, including product mode and performance SLO;
  2. commit the captured conformance fixtures;
  3. complete and review the storage spike;
  4. implement and pass focused, full, race, and failure QA;
  5. update the S3 compatibility reference and operational guidance;
  6. commit and merge the source change;
  7. build and identify the release artifact or container image;
  8. canary a rolling upgrade and observe keyless-drain telemetry;
  9. enable strict mode only after its gates hold;
  10. verify the live endpoint before closing #79.

Passing an earlier gate is not evidence that a later gate happened.

Final recommendation: fix it, but do not rush it

Leaving the current endpoint indefinitely is the wrong trade-off. This is not an obscure response-field mismatch: it affects discovery and cleanup of unfinished data, returns successful but false answers, and changes behavior across nodes and restarts. Those properties undermine the practical meaning of S3 compatibility.

At the same time, a direct implementation patch is also the wrong trade-off. The current disk layout cannot identify legacy uploads, a correct scan needs erasure-aware discovery and quorum, and wire-correct pagination changes observable client behavior.

The balanced decision is:

  • GO for the ADR, AWS fixture capture, metadata-plus-scan prototype, and performance/failure spike;
  • GO conditionally for Option 2 after the product SLO and legacy response policy are approved;
  • NO-GO for a cache-only repair, an immediate durable secondary index, strict-by-default behavior in a patch release, or closing the issue before rolling-upgrade reachability is demonstrated;
  • if implementation capacity is unavailable, GO for an explicit documented divergence and stable error behavior rather than continuing to fabricate successful listings.

This preserves compatibility discipline without pretending that a high-risk distributed listing change is a two-line bug fix.