SILO at One Year: What Actually Changed Since Upstream MinIO?

On October 15, 2025, MinIO shipped its final public open-source release, RELEASE.2025-10-15T17-29-55Z (commit 9e49d5e7a648). Over the next four months, upstream placed the repository into “maintenance mode,” deleted the minio/console repository outright, and in February 2026 archived the main server repository—routing all subsequent CVE patches and bug fixes exclusively to its commercial AIStor product.

On October 26, 2026, SILO tagged RELEASE.2026-10-26T00-00-00Z (commit c3ad836aed0d). Across 376 days, the server repository accumulated 604 commits, evolving alongside its companion repositories silo-console, mcli, and silo-pkg.

A question we hear often from operators evaluating the fork is straightforward: Over the past year, did SILO merely rebrand the binary and apply surface-level CVE patches, or did it fix substantive storage and protocol bugs? Compared with the final upstream MinIO release, what actually changed?

Here is the short answer: SILO preserved MinIO’s erasure-coding storage layout (xl.meta v1.3), HighwayHash bitrot protection, and fundamental read/write quorum rules, while focusing a full year of engineering on security boundaries, S3 protocol and checksum correctness, replication and lifecycle convergence, failed-write rollback and multi-pool safety, and a restored administration console and CLI.

Equally important are the boundaries that have not changed:

1. No distributed fencing, and no claim that every consistency problem is solved. Late commits after lease expiration or network partitions remain possible; strict generation read refusal, automatic cross-pool read recovery, public persistent ListMultipartUploads scanning (#79), access-frequency pool tiering, and multi-pool overwrite affinity (#245) were tested and withdrawn or deferred when their capacity or availability costs proved unacceptable.
2. No head-to-head full-cluster throughput benchmark between the two tags. We record targeted microbenchmarks and amortized overhead checks, not synthetic cluster-wide throughput claims.
3. On-disk compatibility does not mean arbitrary mixed-version operation or rolling downgrade. Durable IAM revocations (siloParentRevocation) and replication metadata tombstones require coordinated upgrades across participating nodes.

The sections below walk through each area in detail, with exact commit references, design records, and operational boundaries.

1. Baseline and Product Stack: What Changed and What Stayed the Same

Our comparison uses two fixed points:

• Upstream baseline: minio/minio RELEASE.2025-10-15T17-29-55Z (commit 9e49d5e7a648f00e26f2246f4dc28e6b07f8c84a, October 15, 2025). Note: between that final release and archiving the repo in February 2026, upstream merged 9 unreleased commits (such as POST policy tag constraints in fa18589d1c and conditional PUT read-quorum adjustments in 18f97e70b1), which never shipped in an upstream release and were addressed independently in SILO.
• SILO target: pgsty/silo RELEASE.2026-10-26T00-00-00Z (commit c3ad836aed0d8a7d23c2e44a5eaa3009d08525e3), released alongside mcli / KES 20261026, Console v2.5.1, and silo-pkg v3.15.2. See the release notes and component matrix.

Across 9e49d5e7a648..c3ad836aed0d, pgsty/silo contains 604 reachable commits spanning feature and security repairs, regression suites, packaging, and documentation. The sections below organize those changes by delivered runtime behavior.

The Four-Component Coordinated Stack

Because the server embeds the Console frontend and imports mc administration packages alongside shared policy and encryption libraries, maintaining the server alone is not enough. SILO maintains a coordinated four-component product graph (plus optional external KES):

Foundations Intentionally Preserved

So existing MinIO deployments can switch to pgsty/silo in place without data migration or client changes, SILO keeps the core storage and wire contracts unchanged:

• Erasure coding and bitrot protection: Reed-Solomon erasure coding and per-block HighwayHash256 verification remain unchanged.
• Read and write quorum rules: Read quorum remains $K$ (data shards); write quorum remains $K$ when data and parity shard counts differ, and $K+1$ when data shards equal parity shards (for example, 16-drive 12+4 uses R12/W12, while 8+8 uses R8/W9).
• On-disk format and system namespace: xl.meta stays at version 1.3 and .minio.sys retains its layout, allowing existing drives to mount directly.
• Wire and configuration compatibility: MINIO_*, MC_*, and CONSOLE_* environment variables, /minio/health/* probes, /minio/v2/metrics/* and /minio/metrics/v3 scrape endpoints, and X-Minio-* headers remain supported (see the compatibility overview and migration guide).

2. Security, Authentication, and Authorization: 7 CVEs and 18 Security Boundary Hardenings

Because SILO retains the github.com/minio/minio Go module path while upstream advisories publish open-ended affected version ranges, automated scanners frequently flag SILO for every historical MinIO CVE.

To separate fact from scanner noise: vulnerabilities such as CVE-2025-62506 (service-account privilege escalation), CVE-2025-31489, and CVE-2025-27414 were already fixed in upstream’s 2025-10-15 baseline. By contrast, the 7 CVEs below were disclosed after 2025-10-15, left unpatched in upstream open-source MinIO, and fixed and verified in SILO:

18 SN-2026-* Security Boundary Hardenings

Beyond public CVEs, adversarial code review and multi-node testing uncovered 18 additional security boundary defects tracked in our Security Advisories ledger:

1. SigV4 SignedHeaders Integrity and Policy Condition Isolation (SN-2026-011, SN-2026-003):
   • Upstream allowed callers to attach unsigned X-Amz-* operational headers (such as governance bypass, tagging, or checksum headers) to an otherwise valid SigV4 request. SILO requires every X-Amz-* operational header present on the request to be covered by SignedHeaders.
   • Upstream populated IAM policy condition maps directly from client headers and query parameters, allowing callers to spoof server-authoritative condition keys such as s3:prefix, s3:versionid, s3:ExistingObjectTag/*, or aws:PrincipalType. SILO isolates server-authoritative condition keys in silo-pkg and binds s3:VersionId checks to the actual target version in DeleteObjects and GetObject.
2. Bucket-Level Write Boundaries, Version-Deletion Authorization, and Admin Action Splits (SN-2026-004, SN-2026-005, SN-2026-009 and 010, SN-2026-017):
   • Tightened resource ARN matching for 12 bucket-level write actions (s3:CreateBucket, s3:DeleteBucket, s3:PutBucketPolicy, s3:PutLifecycleConfiguration, etc.) so object wildcard grants (arn:aws:s3:::bucket/*) or bare arn:aws:s3::: ARNs cannot mutate bucket configurations.
   • Require s3:DeleteObjectVersion for permanent deletion of an explicit version ID, including null; an ordinary delete that creates a delete marker still uses s3:DeleteObject. Batch deletion checks each entry independently.
   • Split user status toggling out of admin:CreateUser into admin:EnableUser / admin:DisableUser; separated self-service password changes into admin:ChangeMyPassword (see password-permission migration); and enforced parent-account ownership checks across IAM admin access-key creation, inspection, and listing endpoints.
3. Service-Account Delegation Ceilings and Object Lock Governance Bypass (SN-2026-015 and 016):
   • Fixed inline session-policy evaluation so parent Deny rules are never dropped when the inline policy omits a matching Allow, and re-verified live parent privileges on temporary STS and service-account sessions.
   • Required x-amz-bypass-governance-retention: true when PutObject overwrites a GOVERNANCE-locked object, and validated retention XML payloads before evaluating Object Lock governance bypass in PutObjectRetention.
4. Durable IAM Revocations, Policy Reload Lock Isolation, and LDAP Long DNs (SN-2026-013):
   • Previously, in multi-node and site-replicated clusters, delayed site-replication events or missed peer notifications could resurrect deleted users, policies, or service accounts. SILO persists deletion tombstones, revision numbers, and parent revocation cutoffs (siloParentRevocation) across both object-backed IAM and STS enumeration, and reloads committed state on peer delete notifications.
   • Moved slow IAM policy loads and KMS decryption outside the hot authentication-cache lock while propagating KMS request cancellation (design record); fixed LDAP StartTLS precedence and hex-escaped long DN path-component limits (design record).

3. S3 Protocol Completeness and End-to-End Checksums

On the S3 protocol surface, SILO focused on three goals: implementing stubbed-out bucket APIs, aligning multi-pool conditional write semantics, and ensuring end-to-end plaintext checksum and size accuracy across multipart, encrypted, and compressed objects.

4. Replication and Lifecycle Correctness

In clusters running active-active replication or remote tiering, the most damaging bugs are silent divergence between sites or premature deletion of tiered data. Over the past year, SILO fixed six replication and lifecycle failure modes:

SILO also isolated per-job and per-target cancellation contexts and fixed ReplicatedCount / FailedCount accounting in replication resync (Replication Resync Correctness), and added per-bucket locks plus optional field-level tombstones (MINIO_SITE_REPLICATION_METADATA_TOMBSTONES) for bucket configuration convergence (Bucket Metadata Convergence).

5. Storage Mutation, Rollback, and Multi-Pool Safety

In an erasure-coded storage engine, the happy path is straightforward; edge-case bugs hide at the intersection of slow drives, partial-write rollbacks, concurrent overwrites, and multi-pool scaling. The 20261026 release ships three groups of storage-layer safeguards:

1. Commit Error Classification and Overwrite Rollback RPC Forwarding

• Missing staging xl.meta classified as 503 SlowDownWrite: When a slow drive or concurrent cleanup causes a staging xl.meta file in .minio.sys/tmp to go missing during commit, SILO classifies the condition as errFileNotFound and returns HTTP 503 SlowDownWrite so clients can retry safely instead of receiving a false 404 ObjectNotFound (error classification design).
• Overwrite rollback option forwarding and timeout boundaries: SILO forwards storage RPC deletion options (including the previous data directory needed for overwrite rollback) while preserving existing drive/RPC deadlines, request cancellation, object locking, read quorum, and healing rules. Note that a client-visible timeout does not mean underlying disk or RPC I/O has stopped or that the write failed to commit—late commits or delayed UndoWrite calls can still occur after partial failures, lease loss, or network partitions (the admitted-I/O draining waits and per-drive mutation guards explored in the October maintenance draft were withdrawn in dcc2f20ba; see the final maintenance boundaries).

2. Multi-Pool Locking, Two-Stage Bulk Deletion, and Migration Safety

For multi-pool deployments:

• Shared cross-pool namespace locks: Fixed pool-local lock isolation where concurrent writes, deletes, and metadata updates on different pools failed to serialize against one another.
• Two-stage multi-pool deletion (#257): Previously, multi-pool DeleteObjects issued parallel deletes across all pools; if the pool holding the newest version succeeded while a pool holding an older copy failed due to a drive error, the stale copy resurfaced as the current object. SILO now deletes in two stages—cleaning non-authoritative duplicate pools first and removing the authoritative copy only after stale copies are gone, while protecting referenced remote tier objects on partial failure (Multi-Pool Delete Safety).
• Rebalance and decommission safeguards: Preserved object tags and revision timestamps across pool migration and prevented decommission from overwriting newer versions or dropping delete markers on the destination pool (Multi-Pool Object Consistency).

3. LIST Recovery Under Drive Disagreement (#239)

During rolling restarts with concurrent overwrites, minority drive disagreements or unresolved entries could cause ListObjects to omit readable keys. While keeping the lock-free fast path intact, SILO now confirms eligible unresolved latest-object entries on demand, distinguishes real deletions from drive read errors, and propagates terminal errors through pagination (LIST Recovery Design Record).

6. Restoring the Full Control Plane: SILO Console v2.5.1 and mcli 20261026

For day-to-day operators, the most visible difference between SILO and upstream MinIO is the administration control plane.

SILO Console v2.5.1: From a Stripped Browser Back to a Full Admin Console

In mid-2025, upstream stripped the embedded community Console down to v1.7.7—removing cluster administration, IAM, policies, lifecycle, replication, diagnostics, and monitoring panels to leave only a bare object browser—and subsequently deleted github.com/minio/console.

Building on the community restoration initiated by Georg Mangold (@gmangold), SILO shipped six major Console releases (v2.0.0 through v2.5.1), restoring the full administration suite while making the frontend smaller, faster, and telemetry-free:

Console 2.5.1 also fixes group membership during status changes, OIDC group permissions, TLS-proxy session cookies, credential exports and keyboard focus. See the Console 2.5 series notes.

mcli 20261026: Honest Exit Codes and Read-Only Checksum Verification

On the client side, pgsty/mc ships as mcli (while preserving MC_* environment variables and ~/.mc / ~/.silo configuration compatibility) and fixes long-standing CLI reliability gaps:

• Read-only mcli checksum verify: Verifies end-to-end FULL_OBJECT checksums across buckets, prefixes, or object versions directly against the server without mutating objects or writing local files, supporting --versions, --Algo, and --json.
• Nonzero exit codes on partial or permission failures:
  • mcli admin info --json now exits nonzero when access is denied or a node reports status: error (upstream exited 0);
  • mcli mirror, mcli mv, mcli retention, mcli legalhold, and mcli sql all return nonzero exit codes on partial object failures, source-deletion errors, or permission denials;
  • Downloads interrupted mid-stream fail immediately rather than reporting a truncated file as complete.
• Safe mirror delete propagation and --dry-run restarts: Prevented mirror --remove from deleting destination objects when source listing fails; added --dry-run to mcli admin service restart; and honored optional persisted region settings in alias configurations.

The 20261026 client also isolates cached identities, drains resync JSON through terminal states, fixes rate overflow, returns nonzero for partially failed start/cancel operations, and adds batch pending output with bounded fallback when live metrics are missing. See the mcli changes and script migration notes.

7. Operations, Configuration, and Supply-Chain Delivery

Across server operations and artifact delivery, SILO made four practical changes:

1. Enforced request-header timeouts and trusted-proxy boundaries:
   • Upstream defined --read-header-timeout / MINIO_READ_HEADER_TIMEOUT without wiring it into http.Server, leaving HTTP/1 connections vulnerable to slowloris-style trickle headers. SILO wired the setting into the HTTP server with a 30-second absolute HTTP/1 header deadline (design record).
   • Added MINIO_API_TRUSTED_PROXIES and MINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIES so X-Forwarded-For is never trusted blindly by default; added CONSOLE_WS_MAX_MESSAGE_BYTES and CONSOLE_WS_MAX_DECOMPRESSED_BYTES to bound embedded Console WebSocket frames.
   • Added a native silo healthcheck subcommand for shell-free distroless containers and tightened MINIO_CONFIG_ENV_FILE parsing to reject malformed lines.
2. Hardened event notification targets:
   • Fixed NATS reconnect handling and JetStream async reply subjects, AMQP confirm-mode lock contention, PostgreSQL/MySQL TLS configuration loading, and preserved Kafka’s default protocol version 2.1.
3. Permanently removed self-update and call-home:
   • Removed in-place binary self-update (MINIO_UPDATE) and all SUBNET registration, telemetry, license checks, and phone-home endpoints. Upgrades are managed cleanly through OS packages (RPM/DEB/APK) and container images.
4. Rebuilt a verifiable multi-architecture supply chain:
   • Built all components with Go 1.27.3 (including built-in crypto/x509 distrusted-intermediate checks), UBI 9.8 and distroless container variants, static curl 8.23.0, and Red Hat’s RHSB-2026-012 root CA blocklist.
   • Published Linux (amd64/arm64/ppc64le/s390x), macOS (amd64/arm64), and Windows (amd64/arm64 ZIP) archives, RPM/DEB/APK packages, and multi-arch Docker Hub images (pgsty/silo, pgsty/silo-console, pgsty/mc, pgsty/kes) backed by GPG signatures, SPDX SBOMs, and GitHub Sigstore build attestations (see Downloads).

8. Honest Boundaries: Withdrawn Experiments and Open Limits

Maintaining storage software in production requires knowing what to roll back when an idea does not survive real capacity or failure testing. Over the past year, we implemented, benchmarked, and deliberately withdrew or closed four designs:

1. Withdrew public persistent ListMultipartUploads disk scanning (#79):
   • Goal: Upstream’s ListMultipartUploads reads only a node-local in-memory cache, losing visibility of incomplete uploads across restarts. In September we implemented a persistent scan across storage pools and erasure sets.
   • Why we rolled it back: Capacity testing showed scan latency grew linearly with prefix depth and abandoned upload count, timing out at 20,000 incomplete uploads and amplifying drive I/O. In Server 20261018 we withdrew public persistent scanning and retired the optional strict mode, returning to the node-local cache plus exact-key lookup (see ListMultipartUploads Design and Capacity Evidence).
2. Deferred strict generation read refusal and automatic cross-pool read recovery:
   • Goal: When partial overwrite failures left split object generations across drives or pools, we tested refusing divergent reads or auto-recovering across pools during GET.
   • Why we deferred it: Fault-injection testing showed that refusing reads after a failed partial overwrite broke MinIO’s established availability contract (where the prior readable version remains accessible). Those strict diagnostics remain gated behind the non-default consistency_strict build tag while production builds preserve existing read-quorum and healing behavior.
3. Closed multi-pool overwrite affinity without merge (#245):
   • Goal: In multi-pool clusters, unversioned overwrites can land on a different pool and leave stale copies on the old pool. PR #245 routed overwrites back to the object’s existing pool.
   • Why we closed it: When the historical pool lost write quorum due to failed drives while another pool remained healthy, forced affinity caused overwrites to fail outright—and two rounds of multi-node m3 benchmarks showed small-object overwrite latency still missed our bar (see Overwrite Pool Affinity).
4. Removed experimental access-frequency intra-cluster tiering (ilm access_*):
   • The experimental feature never met production reliability requirements and was removed in Server 20260916 (see Access-Frequency Tiering Removal).

Remaining Operational and Upgrade Boundaries

When evaluating or upgrading to SILO, keep these boundaries in mind:

• No distributed lock fencing: If a node loses its lock lease during a GC pause, network partition, or stalled drive, an already-admitted commit on that node can still land late; partial metadata updates can still briefly expose stale tags or attributes. The release retains existing timeout semantics; a returned request is not proof that underlying I/O has stopped.
• No unverified cluster-wide throughput claims: We verify hot-path microbenchmarks (for example, silo-pkg v3.14.3 reduced ContainsAdminAction from 24 allocations and 742.5 ns/op to zero allocations and 5.53 ns/op), but we have not run a head-to-head multi-node benchmark between 2025-10-15 and 2026-10-26 on dedicated hardware.
• Coordinated upgrades for new IAM and replication state: While on-disk xl.meta v1.3 remains compatible, features such as durable IAM revocations (siloParentRevocation) and site-replication metadata tombstones require all participating nodes to upgrade together rather than running mixed versions indefinitely or performing rolling downgrades (see the Component Matrix and IAM Upgrade Guide).
• Known minor items: The repository’s Helm chart (7.0.3) default server image tag still points to RELEASE.2026-09-16T00-00-00Z (override image.tag in values.yaml when deploying newer releases), and two legacy batch-lifecycle handlers in Console (batch expiration and batch transition creation) have early-return/empty-slice quirks that do not affect single-rule configuration and are queued for the next Console patch.

Closing Note

When we forked MinIO a year ago, the goal was simple: keep a production-grade open-source S3 server alive with timely CVE patches, a complete administration console, honest bug fixes, and clean binary and container distributions.

One year later—604 commits, 7 CVE patches, 18 security boundary hardenings, and dozens of root-cause write-ups in our Design Archive and Security Archive—that is the ledger of what SILO has done.

If you are running open-source MinIO, see the Migration Guide and Downloads page. If you hit a reproducible protocol, replication, or storage bug in production, open an issue at pgsty/silo.