SILO 20261109: Lifecycle, Object Lock and Batch Authorization

SILO Server 20261109 is now released. Tag: RELEASE.2026-11-09T00-00-00Z; package version: 20261109000000.0.0; source: d998292387cc. It consolidates all Server maintenance since Server 20261026 alongside embedded Console 2.7.0, mcli 20261109, and silo-pkg 3.16.0.

GitHub Release · Full Changelog · Upgrade Guide · Downloads

Components

The component matrix records exact module versions. Updating standalone Console, mcli, or silo-pkg does not replace components inside an older Server binary. KES remains a separate, optional external key service with its own release cycle. The maintained PGSTY stack is the supported target; upstream MinIO/MC compatibility remains best effort.

Lifecycle, pools and remote tiers

• Lifecycle across pools: Recheck lifecycle expiry under the object lock against the key’s authoritative version history and Object Lock state across all pools. Refuse deletion when history or lock protection lacks authoritative quorum; decommission and rebalance move versions whose expiry is refused. Approved version expiries remove every pool copy.
• Exact-key deletion: Restrict ExpiredObjectAllVersions, DelMarkerExpiration, and batch retainVersions: 0 deletion to the exact key, preserving keys under the same prefix. Refuse whole-key purges that contain tiered or Object Lock protected versions. See exact-key deletion.
• Remote tiers and restored copies: Expire only the local restored copy when --days elapses without deleting the underlying tiered version. Preserve remote payloads while another pool still references them, and exclude internal tier-cleanup free versions from visible version history and lifecycle decisions.
• Rule execution and accounting: Apply lifecycle rules whose sole action is DelMarkerExpiration, report only removals that occurred, and queue healing when corrupt history prevents expiry. Lifecycle-generated delete markers and version purges are not replicated, matching S3 lifecycle semantics.
• Pool migration: Move versions across differing erasure layouts, migrate ILM-expired current versions during rebalance and decommission, continue decommissioning remaining buckets when one bucket fails while keeping cancellation active, and prevent rebalance stop panics when rebalance metadata is absent.

See multi-pool lifecycle and restored-copy expiry. Upgrading does not recover objects or remote payloads deleted by an earlier release.

Native restore, locking and reads

Native restore downloads remote-tier data outside the object write lock, then validates ownership and publishes the restored copy under that lock. This eliminates multi-pool restore hangs, wrong-version targeting, and broken ordinary multipart-completion retries. Failed or canceled restore attempts try synchronous marker cleanup for up to 100 ms before handing off remaining work to bounded background retries, avoiding a 60-second lock wait. See two-phase restore.

Single-pool writes and deletes lock in the object’s own erasure-set domain. CopyObject resolves normalized key aliases without deadlocking and waits on a contended source without holding the destination lock; ordinary bulk deletion groups run concurrently across erasure sets. Multi-pool CreateMultipartUpload evaluates If-Match and If-None-Match against the logical current object across pools.

Known partial drive views of an existing object return a retryable 503 instead of a false 404 NoSuchKey. Heal dry runs (mcli admin heal --dry-run) preserve dangling data and propagate the dry-run flag to remote peer bucket heals. Failed writes remove empty object directories, and xl.meta checksum failures are classified for healing.

Object Lock, IAM and batch authorization

S3 DeleteObjects rechecks physical version removals inside the mutation lock used for retention and legal-hold updates, merging protection across pools and serializing against concurrent retention or legal-hold changes (6ef3a52fc). Directory null versions are checked before bulk deletion as well. Replicated deletes preserve the target’s legal hold and retention: a source purge that bypassed GOVERNANCE retention does not grant bypass at a target copy that still carries active retention, so replication refuses the purge and retries.

Batch expiration, key rotation, and replication enforce the authenticated submitter’s local S3 data permissions throughout execution and leave Object Lock protected versions intact. Multi-pool key rotation updates the authorized copy without rendering other pool copies unreadable. Revoked or re-bound identities terminate the job; transient IAM or authorization outages pause execution with a recorded reason and bounded retry. Startup discovery retries storage outages, completed expiration jobs stay completed across restarts, and replication jobs resume scans. Inspect mcli batch status to distinguish complete, complete-with-skips, paused, and failed.

External authorization plugins must return HTTP 2xx with an explicit boolean decision (result or result.allow); missing or null decisions, trailing JSON, and responses larger than 1 MiB are rejected, and each call is bounded to 1 minute. Verify custom authorization plugins before upgrading.

IAM rejects incomplete policy loads, keeps cached authentication available during lazy loads, and applies user edits, service-account mutations, and LDAP group refreshes directly to persisted records so stale caches cannot restore rotated credentials or resurrect deleted identities. Long-DN LDAP login on erasure sets, root Console login with LDAP enabled, and peer webhook target reloads (logger_webhook, audit_webhook, audit_kafka) are also repaired.

Console, client and dependencies

Embedded and standalone Console 2.7.0 brings the 2.6 folder and bucket usage scans and optional PostgreSQL worker into SILO Server, adds ten UI languages with Arabic RTL, fixes folder-size calculation and decryption-key copy over plain HTTP, adds --certs-dir CA trust and listing-only scan budgets to console usage-worker, persists pending bucket changes on worker shutdown, rejects malformed multipart uploads without panicking, and routes /api/v1/docs to Redoc 3. Stop older usage workers before starting 2.7.0 workers; PostgreSQL is required only for that optional worker.

mcli 20261109 fixes destructive rm --purge --dry-run, multi-source alias reuse, self-moves, multipart od downloads, mc cors, MC_REGION / alias region signing, metadata handling, and partial-failure exit codes.

silo-pkg 3.16.0 updates go-jose/v4 to v4.1.9, cpuid/v2 to v2.4.2, and yaml/v3 to v3.0.6. Server also selects gjson v1.20.1, NATS jwt/v2 v2.8.3 and nkeys v0.4.17 (rejecting identity-point and small-order public keys), Google Cloud auth v0.24.3 and oauth2adapt v0.3.1, golang.org/x/net v0.62.0, and pinned golang.org/x/crypto 2f1f834330b2. Standard and distroless container images share the same UBI CA trust bundle (/etc/ssl/certs/ca-certificates.crt) and include mcli legal notices under /licenses/mcli. Helm chart 7.0.4 defaults to SILO and mcli RELEASE.2026-11-09T00-00-00Z.

Upgrade and rollback

1. Stop batch submissions and drain or cancel unfinished jobs before changing versions. Unfinished legacy jobs require resubmission after every node upgrades; completed jobs remain completed. Submitter-bound manifests and progress reports use a format that 20261026 rejects: do not downgrade with unfinished new-format jobs pending. Follow the batch upgrade checklist.
2. Drain incompatible single-pool layouts. If erasure sets in a single pool do not all share the same hosts (18 or more hosts with automatic layout), stop every old process before starting new ones. Old and new delete/copy lock domains do not exclude each other during a rolling restart on this layout.
3. Pause native restores during mixed-version windows. Upgraded nodes return 503 to POST ?restore while any peer is old or unreachable. Full multi-pool lifecycle and bulk Object Lock protection also require all serving nodes to run 20261109.
4. Verify S3 data-path access through every node after restart. silo healthcheck ready checks the HTTP listener and configured KMS/etcd without waiting for storage or IAM initialization, and cluster quorum can precede remote-drive reconnection. Run bounded signed writes and readback through every node before resuming applications, and preserve peer bootstrap (publishNotReadyAddresses: true on Kubernetes). See healthcheck probes.
5. Set image.tag explicitly when reusing Helm values. helm upgrade --reuse-values keeps the previously saved tag; pass --set image.tag=RELEASE.2026-11-09T00-00-00Z.

Costs and limits

Safety checks read additional cross-pool metadata and refuse operations that cannot be proven safe. Physical version purges in Object Lock buckets execute sequentially through the single-object path during DeleteObjects, so large purge batches take longer (ordinary buckets and delete-marker creation retain the bulk path). In a bounded Linux ARM64 two-pool check, the average per-round median for 1,000 removable locked-bucket versions increased from ~202 ms to ~310 ms; steady HEAD/GET medians changed by ~+1.5% / +3.4%, and alternating 4 KiB / 1 MiB write rounds stayed within normal run-to-run storage variance without extra request-path syscalls. These VM directory measurements document expected safety trade-offs rather than production disk throughput.

Storage timeouts still do not prove that a disk write did not commit, and this release does not add distributed fencing or recover previously deleted data. Known open limitations remain in ListMultipartUploads prefix/pagination (#79), multi-site delete-marker purges (#217), replayed bucket creation under site replication (PR #241), tag-only read authorization returning 403 when the object is unreadable (#293), and upstream SDK multipart-completion retry headers (mcli put --if-not-exists; see multipart completion retries).

Delivery and verification

Release commit d998292387cc passed Go CI, VulnCheck, Test Release Pipeline, and the tagged release build, along with two-pool and four-node runtime checks, S3/client/Console integration, and object/IAM upgrade and rollback checks.

The GitHub release publishes 36 Server assets (Linux/macOS tar.gz and Windows ZIP archives, x86_64 and arm64 RPM, DEB, and APK packages, SHA-256 checksums, SPDX SBOMs, Sigstore bundles, and provenance attestations), alongside 44 Console assets, 19 mcli assets, 19 KES 20261026 assets, and the source-only silo-pkg module. Multi-architecture container images (linux/amd64 and linux/arm64) are published on Docker Hub:

• docker.io/pgsty/silo:RELEASE.2026-11-09T00-00-00Z (latest): sha256:08ce1505b5eacefbad1ca07159b242b9ce045a11c2817e3bc257b25f77bbc02d
• docker.io/pgsty/silo:RELEASE.2026-11-09T00-00-00Z-distroless (distroless): sha256:fdf726815e38c2d1cb06c640e237daa83b3914e58099aecd073fef9d8e42cb00

Thanks to @botdeniska for the storage, lifecycle, restore, and IAM reports and fixes, @dihmandrake for heal dry-run safeguards, and @haf for the webhook reload report. Console and client credits are recorded in their respective release notes and the contributor record.

Source Changelog · Changes Since 20261026