SILO 20260916 Release Notes

Version date: 2026-09-16 · Published: 2026-09-16 UTC (September 17 in China) · Tag: RELEASE.2026-09-16T00-00-00Z · Package version: 20260916000000.0.0. Use the download page for installation.

The source range runs from released commit 9b11dc9469e6 to released 2a4d51406b7e: 210 commits, including 140 non-merge commits. These notes describe the final behavior, including the listing-quorum, version-purge and migration-tag repairs after #214. Changes already present in the previous release, such as Go 1.27.1, are not counted again; withdrawn features are described by their final state.

Highlights

• Stronger request authentication: reject unsigned operation headers in ordinary and presigned SigV4, verify presigned body hashes, and use authenticated inputs for signature-age and payload-hash policies.
• Durable IAM revocation: retain deletion revisions, parent revocation boundaries and membership grant times across delayed events, same-name recreation and restart/recovery.
• One logical object across pools: coordinate conditional PUT, multipart completion, conditional DELETE, addressed-version deletion and metadata updates under consistent selection and locking rules.
• Preserve tags during pool migration: rebalance and decommission retain tags, their original revision and ordered empty states for ordinary and multipart objects.
• Correct encrypted and federated copies: preserve raw SSE-C replicas, plaintext part sizes, independently ordered lock/tag metadata, and the checksum, version and time of the actual destination write.
• More reliable replication and configuration convergence: repair tag resurrection, marker purge/MRF recovery, resync counters/cancellation, and bucket metadata source timestamps and deletions.
• Opt-in durable multipart discovery: add persistent scanning, global pagination and read-only preflight. Legacy remains the default; capacity and late-creation-write limitations remain explicit.
• Updated administration stack: select Console 2.4.1, mcli 20260916 and pkg 3.14.1, including bounded browsing, streaming ZIPs, reliable failure exits and policy correctness fixes.
• Two upgrade requirements need advance preparation: password permissions change the meaning of existing policies, and durable IAM revocation requires coordinated upgrades and complete backend backups.

Versions and components

The release explicitly replaces Console and MC with v0.0.0-20260916075814-1360e26d976d and v0.0.0-20260916070421-e952aa78f10a. Go does not inherit a dependency’s replacements. The shared package uses the PGSTY module path directly; Server, Console and MC retain their historical module identities and protocol names. minio-go/v7 continues to use a verified upstream revision.

Server 20260916 explicitly selects this dependency graph. Installing a standalone component does not replace anything already compiled into a Server binary or image. The supported release stack is SILO + Console + mcli + silo-pkg; compatibility with unmodified upstream MinIO/MC remains best effort.

Security and authorization

This cycle includes four local security identifiers. SN- identifiers are project tracking IDs, not CVEs. See the signed-header contract and September security notes.

Related changes remove ambiguous duplicate replication-source headers, derive s3:signatureAge from the signed date rather than a scratch header, align presigned query/header payload-hash precedence, and inject the server-generated PutObjectTagging header after authentication but before authorization. X-Amz-Content-Sha256 may remain outside SignedHeaders because its effective value is separately bound and checked. Explicitly signed UNSIGNED-PAYLOAD retains its meaning.

This does not claim complete unsigned-header coverage for streaming SigV4 seeds. Unsigned-header rejection retains 400 AccessDenied, rather than changing to AWS’s 403. The Console fix adds no global sharing-disable variable and does not bypass S3 authorization; it should not be described as proven arbitrary-host SSRF.

Password permission split: a compatibility change

Server #175, pkg #6 and Console #51 separate password self-service from user administration:

• An internal user changing their own password checks admin:ChangeMyPassword, allowed unless explicitly denied.
• Creating users or resetting someone else’s password still requires admin:CreateUser. Existing root, STS and service-account restrictions remain.
• A standalone CreateUser Deny no longer locks the caller’s password. An existing ChangeMyPassword Deny now constrains this endpoint.
• Updated built-in readonly no longer overrides a separate CreateUser grant with its own deny; the package also supplies the separate consolereadonly capability policy. Saved policy overrides are not rewritten.

To retain the old combined restriction, deny both actions in the same statement before upgrading, preserving its resources and conditions. Keep both denies during rollback: an old Server cannot enforce a password-only deny on this endpoint. See password migration.

Shared policy correctness

pkg #4 fixes statement deduplication that omitted NotResource from hashing or collapsed distinct string-condition sets, potentially discarding Deny clauses. Directly constructed policies report Deny before indexing. Multi-star wildcard matching no longer uses exponential recursion while retaining byte matching and historical MatchSimple boundaries.

Requests that relied on lost denies may now be rejected. If a clause has already disappeared from stored JSON, upgrading cannot reconstruct it: compare with the original policy and reapply it. These are effective Server changes from upgrading pkg v3.13.2, not merely new dependency numbers.

IAM replication, revocation and recovery

Deletion revisions remain in the original IAM paths for users, service accounts, groups, policies and policy mappings. Normal listing and authorization hide deleted records. Object and etcd backends serialize revision comparison and persistence, retain source timestamps, and prefer deletion over a live record at the same timestamp.

• Delayed same-backend deletion notifications reload committed storage rather than deleting a subsequently recreated identity.
• Recreated parents retain RevokedBefore; new service accounts and internal STS credentials carry a signed siloParentRevocation boundary. Replaying or editing an old child does not renew its authority.
• Each group member retains its own grant time. Editing another member or group status does not regrant every member.
• Newer service-account snapshots can replace an older copy of the same key while preserving secrets, disabled state, absolute expiration and revocation boundaries. Credential-type conflicts fail; expired snapshots cannot extend old credential life.
• Healing resumes after leadership is regained, retains acknowledgements and retries bounded batches. Natural expiration of ordinary STS credentials does not automatically create permanent deletion history; reusable service-account keys retain revisions.
• IAM metrics report retained revocations, healing failures, duration and last success. See the design and metrics.

This is not instantaneous cross-site revocation. A parent deletion may commit before later cleanup returns HTTP 500; that response does not prove old credentials remain valid. Tombstones have no TTL or automatic compaction, and storage/startup costs grow with retained history. Concurrent independent membership edits, equal-time conflicting live updates, arbitrary clock skew and cross-site individual-token revocation retain documented limits.

S3 conditions and multi-pool consistency

The coordination layer selects a logical object under a shared namespace lock before evaluating conditions or merging copies of the same version. Relevant draining pools participate. Retention, legal hold and tags each use their own revision; object modification time is not a version for every metadata field.

Availability tradeoff: unverifiable metadata in another relevant pool can fail conditional PUT/completion or coordinated deletion even when another pool serves GET/HEAD. Read-quorum failures return 503; restore readability before retrying. Cleanup can fail after partial physical mutation; there is no distributed rollback. Pending outbound delete replication retains versions until purge completes, and success does not prove immediate removal from every drive.

A public conditional PUT with a destination versionId compares the current object while preserving the requested write version. It does not clean stale copies, reverse previously accepted overwrites or provide global clock ordering. Multipart #190 and ordinary PUT #207 address separate defects.

Unsupported boundaries: batch DeleteObjects XML ETag still provides no condition protection. Empty If-Match follows ordinary deletion; internal recursive prefix deletion bypasses object conditions. There is no s3:if-match policy key or new GetObject authorization requirement. Use the supported single-object path when conditions matter. See conditional DELETE and multi-pool consistency.

Preserve object tags during pool migration

fced8630365f repairs lost object tags during rebalance/decommission. The read representation stores tags separately in ObjectInfo.UserTags; forwarding UserDefined alone could omit the canonical tag field needed by the destination write.

The four ordinary-PUT and multipart-initialization entry points now rebuild that field in an independent metadata copy. They preserve the original tag revision, including absent/empty revisions and ordered empty states, without generating a new revision or mutating the reader’s map. Writes and multipart completion retain existing rereads under the coordination lock so an old reader does not override a later tag update or removal.

The patch does not recover tags lost during earlier migrations. Audit previously migrated objects and tag-dependent policy and lifecycle rules separately.

Null-version listing quorum

8d06424b1 prevents a newer minority from hiding an older null version that has listing quorum. It recounts per header only when the original selection lacks quorum, every nonempty drive stream has exactly one ordinary null version, and all share the same erasure layout. Mixed version histories retain their previous behavior.

Remaining limitation: #218 tracks successful listings that still omit readable objects during rolling restarts with concurrent overwrites. Do not run synchronization tools that delete destination objects based on such listings; list again after the cluster stabilizes.

Encryption, checksums and object attributes

• Raw SSE-C replicas: store authorized replica ciphertext verbatim, avoiding another encryption/compression pass caused by destination defaults (#109, #122).
• Plaintext multipart sizes: preserve logical part lengths in replicas, repairing part-number/range reads, object size after replacement and attributes (#119, #124).
• No new compressed SSE-C objects: exclude ordinary PUT, multipart initialization, COPY and Snowball SSE-C writes from compression. SSE-S3/KMS retain their encryption-compression options (#118, #126).
• Retransmission repair: update existing or undecodable SSE-C replicas through object retransmission instead of ordinary metadata COPY, preserving newer destination retention/hold (#120, #134).
• Explicit checksums during key rotation: perform the required full rewrite, including requests for the existing algorithm. A multipart source may become a single-part destination and its ETag may change (#113, #123).
• Logical attributes: GetObjectAttributes returns plaintext part sizes for compressed/encrypted objects and terminates pagination correctly with sparse part numbers (#114, #115, #128).
• Honest unreadable-object errors: return 500 for integrity/decryption failures instead of a 206 accepted as a successful read (#110, #121).
• Chunked checksum interoperability: honor and validate the actual checksum header on non-trailer chunked PUTs even when x-amz-trailer is advertised (#107, #143).

These prevent new errors and provide supported repair paths; they do not rewrite historical encrypted data. Retain keys and exact version identities when auditing compressed SSE-C, wrong part sizes or unreadable replicas. See SSE-C integrity.

Federated CopyObject

This section concerns remote forwarding in legacy etcd bucket federation, rather than bucket/site replication. Changes accumulate through #157, #159, #163, #172, #177 and #179.

Issues: #99, #100, #158, #166–#171. A remote write may have committed before response validation fails; inspect the actual destination version before retrying a versioned copy. See the federated copy contract.

Object Lock, tags and replication recovery

Independent lock and tag ordering

• An ordinary write supplying only legal hold no longer suppresses bucket default retention; ON and OFF do not substitute for retention selection, and explicit timestamp precision is retained (#165, #177).
• Empty-value and timestamp-only retention/hold removals participate in ordering, preserve newer destination state and no longer cause endless retransmission (#111, #117, #129, #131).
• Replica writes reconcile authoritative Object Lock state across pools, extending the earlier single-set boundary (#133, #178).
• Pool reconciliation carries tags with their revisions; SSE-KMS metadata COPY and SSE-C rotation retain tag timestamps. Local PUT/DELETE tagging advances revisions monotonically, and empty tags are ordered deletions rather than missing state (#189, #193, #196).

Both replication endpoints need the fixes to respect empty-tag tombstones. Explicit resync/heal adds a metadata COPY for objects with recorded tag revisions; destination default KMS can make that operation rewrite data. Tag-filtered rules still evaluate target eligibility using the post-deletion empty tags. Missing or malformed historical revisions are not reconstructed. See Object Lock ordering and replicated tag ordering.

Delete markers, MRF and resync

• Correct single-object marker-purge classification and retain marker identity/retry state through workers, scanner/heal and persistent MRF recovery. Accept a relevant 405 marker response only when bucket, key, version and modification time match. Normalize purge audit status from COMPLETE to COMPLETED (#153, #162, #196).
• Expose bounded MRF queue drops through administration, metrics and deduplicated logs, and align object/delete worker allocation. This does not turn the bounded queue into a lossless unbounded queue (#152).
• Persist final counters before publishing resync Completed; count success by the actual replication outcome rather than the existence of a target object (#136, #139).
• Dispatch only to the resync target ARN. Give each resync ID an independent cancellation context, drain blocked dispatch/walkers/workers, and prevent older work from overwriting terminal state (#141, #137).

Exact-version purges and queued creation

eb4f5e5b3 stops a pure version purge from creating markers on drives that lacked them. Missing-version retries require write-quorum absence; purge results combine removed and reliably absent replies. Receivers resolve the addressed version across pools, and marker healing preserves stored replication and purge metadata.

254b19ac0 rereads queued marker creations under the replication lock. A missing version, non-marker version or version under purge makes the task stale; an uncertain read is retried through MRF. The check covers tasks queued by the DELETE handler, GET/HEAD/LIST healing, scanner and MRF.

358ab38fb reports a delete marker only for a stored marker, preventing a data version awaiting purge from incorrectly returning x-amz-delete-marker: true and producing the wrong removal event.

Remaining limitations: creations already in flight, replays from another site, and minority residue after a crash following a majority-acknowledged purge remain tracked in #217. This is not a persistent receiver-side purge fence.

Replica metadata normalization

Restore only six replication-specific fields after ordinary metadata extraction, preventing transport-only aws-chunked from becoming stored Content-Encoding. Trusted Snowball entries no longer inherit ordinary metadata from the outer archive (#185, #194).

These changes do not scan/rewrite existing objects or recover lost tags, lock history or every unfinished purge. The read-only audit tool and runbook cover exact versions, mixed encodings, ambiguous findings and controlled recovery. Blind bulk self-COPY is not a general repair procedure.

Bucket configuration, caches and startup

PR #180 completes source-state convergence for #77. Policy, Tags, SSE, Quota, Versioning and Object Lock read/compare/mutate/save under the same metadata.lock. Peer events retain source time; local changes advance monotonically past retained time. Equal-time deletions win, and stable content keys resolve other conflicts.

• Dedicated events, bulk, import, initial synchronization and heal use the same rules. Omission is not deletion; empty Versioning/Object Lock events still do not delete those settings.
• Canonical policy JSON allows accepted NotAction/NotResource policies to be retrieved/exported. Statement/set order may change without changing authorization.
• Empty Policy retains deletion semantics. Zero Quota remains a live document imposing no capacity limit, rather than silently becoming a different deletion state.
• Missing historical bucket Created values can be recovered from physical information and persisted. Adoption does not turn empty defaults into deletions, and unknown/invalid/unreachable sources receive bounded diagnostics.
• MINIO_SITE_REPLICATION_METADATA_TOMBSTONES is off by default. Upgrade every participating site before enabling it consistently and restarting, exposing empty-payload Tags/SSE/Quota deletion times to heal/initial sync. Missed-deletion convergence remains limited while it is off.

This is not a causal clock or automatic recovery of historical receive-time pollution. This field-level compatibility gate does not make the entire release, including new IAM state, safe for rolling downgrade. See bucket metadata convergence.

Other fixes close remaining bucket deletion/update/lifecycle/cache publication races, remove the reload guard that mistook the maximum field timestamp for a whole-record version, and preserve per-bucket CORS through metadata export/import (#105, #112, #127, #156, #164).

#116 adds startup-readiness documentation and Linux restart evidence. An online status or successful health probe does not replace signed PUT/GET canaries at every coordinator. Health endpoints were not redesigned as a complete data-path barrier.

Multipart discovery, pagination and cancellation

PR #198 adds durable discovery; #213 restores conservative defaults and rollback compatibility. The final default is legacy, not the strict default described by the original #198 implementation.

New writers record reserved bucket/key identity in the existing quorum-written xl.meta and remove upload-only fields on completion. There is no secondary index or directory migration. The shared maxUploadsList cap decreases from 10,000 to 1,000 in both modes; this does not establish complete legacy pagination semantics.

Enable strict only through process environment MINIO_API_MULTIPART_LISTING=strict and restart. It is not a dynamic shared configuration key. Invalid values diagnose and fall back to legacy while retaining other API settings. First upgrade every writer, drain legacy uploads, run the read-only GET /minio/admin/v3/multipart-preflight endpoint with admin:StorageInfo, and validate capacity. Unidentified legacy uploads can block strict lists for other buckets; preflight readiness cannot prevent an old writer from creating more.

Each process admits two scans; each scan has 16 identity workers, four metadata workers, a 30-second context and a 100,000 aggregate directory-entry budget. This includes drive duplication, not 100,000 unique uploads. Every page rescans persistent state. Exhaustion or uncertain coverage returns 503 rather than a successful partial page. A temporary 10,000-upload/four-APFS-volume experiment took 18.7 seconds for one page, missing the provisional five-second target; this is not production-scale performance certification.

Both modes fix wrong-key/bucket Abort evicting a valid upload’s cache and peer notification cancellation after unlocking. HTTP retains idempotent 204 for an absent upload; that is not proof of physical drain. Late physical creation writes can still restore an upload after acknowledged cancellation; no durable creation fence was added. #79 therefore remains open. The current enablement and rollback contract supersedes historical proposals.

Cumulative Console 2.4.1 improvements

Beyond sharing and password permissions, the embedded selection includes these post-20260903 changes:

• Cursor browsing: ListObjectsV2 pages of 50/100/250/500/1000 entries, with explicitly page-scoped filtering, sorting and selection; no full-bucket count scan. Failed page changes preserve the current listing, cancellation releases list workers, and rewind discloses row/time limits. Empty/null versioning responses no longer crash the list.
• Exact operation identity: retain bucket/key/version for deletion and constrain directories to their trailing-slash prefix. Lifecycle edits preserve current/noncurrent, date-based and unexposed settings, read legacy prefixes and remove stale filter representations. Replication-rule deletion retains shared targets and uses configuration deletion for the last rule. Governance bypass is explicit and off by default for authorized users, including object-only retention.
• Capabilities and sessions: gate Create Bucket on s3:CreateBucket; preserve sessions after background transport failures; URL-encode Watch filters and preserve structured Inspect errors without consuming bodies twice.
• Embedded compatibility: restore loopback TLS login, apply embedded local-proxy trust only to an explicitly initialized TCP loopback peer, retain standalone trust-none defaults, and forward all four WebSocket limits instead of incorrectly capping public browsing at eight connections per source (Server #108, #147, #148).
• Streaming ZIPs: use a backpressured, cancellable file writer where supported and native download management elsewhere, without buffering whole archives in JavaScript. Native handoff cancellation belongs to the browser. Show percentages only with reliable length and recommend mcli above 5 GiB or when size is unknown.
• Recovery/localization/accessibility: recover from malformed routes, persisted preferences and render errors, and complete English/Chinese labels, icon names, sign-out text and keyboard tooltips. Shared dropdowns no longer select a stale hovered item after a quick click/touch, or select another value when clicking a disabled item.
• Standalone packages: improve reproducibility, immutable tooling/browser inputs, SBOMs, signatures and provenance. Preserve minio-console.service, console-user and /etc/default/console; move certificates to /etc/silo-console/certs with explicit migration instructions. Installation does not automatically restart the service.

References: Console #47, #50, #57, 2.4.1 notes. Standalone releases and public-image acceptance are separate from Server embedding.

Cumulative mcli 20260916 improvements

The bundled client includes both 20260913 and 20260916 changes:

• Pipe honors applied global quiet/JSON settings, avoids progress frames on redirected output, preserves explicit checksums for empty input and streams nonempty input. MC_* booleans accept on/off/enabled/disabled.
• SQL/S3 Select, transfers, mirror permission errors, legalhold, partial recursive retention failures and mv source-deletion failures return nonzero. Mv waits for deletion results without rolling back successfully copied destinations.
• Per-object mirror permission errors do not stop later objects; finite failed jobs exit 1 and do not print an ordinary successful completion summary. Explicit JSON summaries report failure, and very short transfers retain finite speed values. Existing listing/watch cancellation-retry boundaries remain.
• Mirror –remove –watch preserves targets when historical source versions are deleted. Administrative dry runs never fall back to a real restart; non-TTY results remain reliable.
• Empty retention durations and invalid find –regex expressions report errors instead of panicking; JSON usage errors retain valid output contracts.
• The SDK detects CopyObject errors embedded in HTTP 200, preventing mv from deleting its source after false copy success. This applies when the client selects CopyObject, normally below 64 MiB or with –disable-multipart, and does not replace independent validation of the default large-object ComposeObject path.

Scripts must inspect the final exit status rather than initial success messages or progress bytes. Skip-errors does not turn an observed failure into exit 0; some retention JSON failures may lack readable error text. Commands, aliases, MC_* variables, configuration format and package naming remain compatible.

HTTP, TLS and observability

• Absolute header deadline: #196 wires –read-header-timeout / MINIO_READ_HEADER_TIMEOUT into the server and prevents small reads from indefinitely extending HTTP/1 header parsing. The default is 30 seconds. Bodies retain rolling idle timeouts, not a new total transfer deadline. Shorter values constrain TLS handshake reads; HTTP/2 retains its independent behavior. TLS writes and handler CPU/storage waits do not gain a new total deadline. Negative values disable the header-specific limit and are not a recommended routine workaround. The design responds to #183.
• Go TLS defaults: remove eight explicit Server curve overrides so listeners, nodes, identity providers, replication and etcd honor Go 1.27 defaults and applicable GODEBUG controls without weakening certificate/hostname checks (#164). #154 is closed, but code and synthetic OIDC evidence do not establish a retest of the reporter’s hidden deployment. Total discovery/JWKS timeouts remain separate work. See TLS/OIDC.
• CPU metrics race: locking around both map levels read by loadCPUMetrics prevents concurrent resource updates and Prometheus scrapes from terminating the process. The repair entered main through #214. Separate #215 closed as superseded, and issue #210 also closed; #215 itself did not merge.
• Quota/logger metrics: read the effective quota field, make AddSystemTarget idempotent and serialize logger initialization/console-target publication, avoiding duplicate collectors after log subscriptions (#106, #150, #132, #151).

Dependencies, delivery and maintenance

• Upgrade amqp091-go 1.10.0 to 1.14.0, including bounded malicious-frame allocation; SILO reachability requires an AMQP notification target (#176, #177).
• JWX 3.3.0 escapes custom JSON field names; strfmt 0.27.2 repairs Go 1.27 hostname validation; LZ4 4.1.30 fixes reader, partial-read and concurrency behavior, with Snowball/S3 Select integration regressions. Dependency presence alone does not prove application exploitability.
• Cumulative upstream SDK changes include configurable upload limits, streaming Content-Type signing, caller TLS trust for RDMA, consistent listing checksums/optional restore status, and the HTTP-200 CopyObject repair above. These are adopted SDK capabilities, not a new Server RDMA service. See the upstream adoption record. Console also updates React Router to 7.18.4 and retains js-yaml/smol-toml denial-of-service fixes with rebuilt embedded assets.
• Refresh Go x/* dependencies while retaining Go 1.27.1, the NetBSD go-systemd 22.6.0 replacement and tablewriter compatibility selection. Go 1.27.1 already shipped in the baseline.
• Refresh container base digests and build static curl 8.22.0 from verified source on both Linux architectures. Pin published mcli 20260916 archives and SHA-256 values in the installer and image.
• Pin GoReleaser 2.18.1 and action v7.2.3 by full commit; dependency-only changes now trigger Test Release Pipeline. Include Helm Chart 7.0.3 in the tag with the new Server/client defaults; there is no separate chart publication.
• Fix conditional PUT and tag-replication test-fixture races, restore valid integration credentials, and add CPU/tag/multipart/S3 Select race coverage. Test-only repairs are not additional production data fixes.
• Maintain rebrand guards, AGPL-3.0-or-later/DCO/no-CLA contribution rules and human contribution records; refresh README repository cards daily.
• Consolidate formal product, upgrade, security and design documentation on this site. Retain repository entry points and necessary compatibility/testing documents; move temporary investigations out of public source and retain historical evidence through immutable links.

September 4 Helm image-tag correction, mcli archive pinning and recovery container publication completed the previous release’s delivery. They are included in the scan but are not new 20260916 S3 capabilities.

Withdrawn experiment

GET-frequency-based cross-pool ILM tiering #60 was merged during this cycle and removed by #188. It never shipped in Server 20260903 and is not a feature of this release. The tracker, mover, scanner hooks, dedicated configuration/XML actions and metrics are removed. Ten old ILM keys are accepted but ignored; v8/v9 usage caches are readable and v8 is written without retired fields.

Intermediate development deployments should remove access-only rules and retain ordinary actions in mixed rules. Ordinary expiry, remote tiers, rebalance and decommission remain; independent multi-pool lock, conditional-delete, write-coordination and shared-tier protections remain. See migration.

Upgrade and recovery requirements

1. Freeze final identities. Checks on this release SHA do not automatically apply to later source. Verify Server, Console, mcli, packages, images and Chart separately. Official image names are docker.io/pgsty/silo, docker.io/pgsty/mc and docker.io/pgsty/silo-console.
2. Migrate password denies and compare original policies. Preserve CreateUser and ChangeMyPassword in the same Deny when the old combined restriction must remain. Recover previously lost clauses from original sources.
3. Use coordinated maintenance. Pause IAM writes and isolate unknown offline sites. Upgrade every participating node/site, including nodes sharing IAM without site replication. Mixed old/new access to one IAM backend and rolling downgrade are unsupported.
4. Save complete recovery points. Back up IAM storage and required encryption material; mcli admin cluster iam export omits deletion history. Stop relevant processes before restoration so cached peer acknowledgements are not reused.
5. Verify every endpoint. Old revoked credentials must fail; new users, service accounts and newly issued STS credentials must work at every node. Check signed PUT/GET, bytes, versions, tags, retention and replication convergence before reopening traffic.
6. Reconcile revocations after rollback. An old snapshot loses later revocations. Reconcile each restored site and test every endpoint; use differently named replacement identities/reissued credentials when needed. A successful disable command at one site is not global proof. Pre-restore STS continuity is not promised. Do not delete tombstones online or convert only live records to force an old binary to start.
7. Enable optional behaviors independently. Metadata tombstone export requires all participants upgraded. Strict multipart additionally requires all writers updated, legacy uploads drained, preflight and capacity validation. Ordinary upgrades may keep default legacy mode.
8. Handle development-build remnants. If an intermediate build persisted api multipart_listing, back up configuration and use mcli admin config reset ALIAS api multipart_listing to remove only that key, retaining other settings. Do not reset the entire subsystem or replay history containing the key. Follow the separate access-tiering retirement guide if that experiment was used.
9. Audit historical data separately. Inventory unreadable SSE-C, wrong part sizes, tags/locks, aws-chunked pollution, past erroneous overwrites and pending purges. Validate approved repairs on copies; installing the binary does not prove recovery.

Runbooks: IAM upgrade/recovery, historical replica audit, password migration and multipart contract. There is no wholesale object/erasure-layout migration, but new persistent IAM state means old-version object readability is insufficient to certify whole-stack downgrade.

Known limitations and remaining work

Validation and release status

Released source 2a4d51406b7e passed Go CI and the Test Release Pipeline. The Release build and Finalize completed. The GitHub release provides 36 assets, including six-platform archives, Linux packages, SBOMs, checksums and attestations. Earlier local upgrade and recovery records retain their own source identities and validation limits; they do not establish final artifact or production rollout acceptance.

Console v2.4.1, mcli 20260916 and pkg v3.14.1 have independent published releases. At the 2026-09-17 recheck, anonymous token requests for standalone docker.io/pgsty/silo-console returned HTTP 401, so public pulls are not confirmed. Use Console’s GitHub binaries or packages. This standalone image limitation does not affect the source selected for Server embedding.

Issue index and acknowledgements

Fixed or completed within the described scopes: conditions #10/#133/#144/#199; federation #99/#100/#158/#165–#171; bucket metadata #77/#105/#112; encryption/attributes #107/#109–#115/#117–#120; embedded Console #108/#147/#148; replication recovery #136/#137/#139/#141/#152/#153/#185; runtime/metrics #106/#116/#150/#154/#176/#183/#210. The pool-migration tag repair is recorded as a direct main commit; #79 remains partially implemented and open. The sections above link the individual records and limits.

Thanks to reporters, patch authors and reviewers, particularly @mrjavadseydi (quota, multipart and tiering exploration), @nikitapogromsky (logger idempotency), @Aeirx (federated legal hold), @magicxor (early conditional DELETE proposal), @julienlau (purge/MRF), @chodorenko (replica encoding), @orenyomtov (signed headers), @jiri-pejchal (Console sharing), @AEGEGE (HTTP report), @aschyolkin (CPU race), and @vampywiz17, @kh0mka and other reproduction/feedback contributors.

Original PRs #12, #125, #155, #184, #187 and #195 retain their actual closed/superseded status; incorporating work through later implementations does not make those original PRs merged. See the full contribution record.

Previous release · Release source diff · Component matrix