• Design
• Lifecycle
• Tiering
• Data Safety

This record covers issue #281 and PR #282 by Denis Babchuk, evaluated against main at 69fcf50cd. The fix ships in the same maintainer integration PR as the multi-pool lifecycle fix (Multi-Pool Lifecycle Expiry, #278). That PR keeps the contributor commit unchanged and adds the safeguards described below. It also depends on the single-pool lock-domain fix, which must merge with it or before it.

Status: fix approved by review and committed locally. Tested together with the other fixes for the next release on one integration branch: the full local suite with the kqueue,dev tags, the CI race selections run locally and an adversarial review of how the fixes interact all pass. Not merged or released.

Short answer

The report is real. After POST ?restore, a transitioned version gets a temporary local copy until the expiry date in x-amz-restore. When that date passes, lifecycle must remove the copy and nothing else: the version stays transitioned and is read from the tier again.

For a noncurrent version this already worked. For the current version, the cleanup was sent as a delete of the object, because the version ID was not passed:

• a versioned bucket got a new delete marker on top of the object, so GET and HEAD returned NoSuchKey;
• a versioning-suspended bucket had the delete applied to the null version underneath, which removed that version’s data;
• without a null version, the delete failed silently and the restored copy stayed on the drives forever.

A plain GET or HEAD after the expiry date was enough to trigger it.

The core repair is the one PR #282 proposes: the cleanup always addresses the exact version the restored copy belongs to. Review then found that cleanup and restore had other ways to touch the wrong version, so the fix ships with four more groups of changes:

• a versioning-state correction;
• a storage-level guard that lets only completed tiered versions give up a restore cache;
• a set of native-restore repairs, so that restore itself cannot create, overwrite, or deadlock on the version it rehydrates;
• an ownership protocol for native restore, from the adversarial review. Each restore attempt owns the version it rehydrates. A restore stages its copy outside the object lock and commits it briefly under the lock (Native Restore in Two Phases). Every restore decision uses the metadata that a quorum of drives agrees on.

Classification

When it happens

The restore sets its expiry date to UTC midnight at least Days (minimum one day) ahead, so the defect shows up a day or more after a restore. The reporter’s test moves the date into the past and runs each lifecycle path, on one pool and on two:

On main, 32 of the 56 cases in the contributor’s test fail, for every path and both pool counts.

Root cause

Lifecycle.Eval returns DeleteRestoredAction for an expired restore of the latest version and DeleteRestoredVersionAction for any other version. expireTransitionedObject sets the version ID only for actions that DeleteVersioned() accepts, and that set contains DeleteRestoredVersionAction but not DeleteRestoredAction.

For the current version, DeleteObject therefore received no version ID and treated the request as a delete of the object. The ExpireRestored flag that should limit the delete to the restored copy had no version to act on:

• with versioning enabled, the ordinary path wrote a new delete marker;
• with versioning suspended, the request addressed the null version, which xlMetaV2.DeleteVersion found and whose data directory it returned for removal.

Where the behavior came from

Provenance

Upstream origin, as a regression; not introduced by SILO. The defect was introduced by ee158e1 (minio/minio#19093, Krishnan Parthasarathi, 2024-02-22), first released as RELEASE.2024-02-24T17-11-14Z. The restored-copy branch did not always lack the version ID. SILO inherited the code at the fork, and every one of the 18 SILO releases, from RELEASE.2025-12-03T12-00-00Z through RELEASE.2026-10-26T00-00-00Z, contains it. No SILO commit changed the branch.

Before #19093, every restored copy was addressed by the version ID of the object it belongs to, so a restored current version with a version ID was cleaned up correctly. The commit message of #19093 says that it fixes tiered objects whose current version was deleted instead of getting a delete marker. For that it replaced the version ID with the DeleteVersioned() gate, which the non-transitioned expiry path already used, and the gate leaves out DeleteRestoredAction. The restored-copy branch sits in the same function and lost its version ID as a side effect. A restored null current version reaches the expiry with an empty ID, and an empty ID was never addressed correctly, before or after #19093.

What SILO changed around the branch.

The nine lines from DeleteRestored() on are identical in all 18 SILO releases and in upstream master, and so is DeleteVersioned(). The fix therefore touches code that SILO has never modified.

The native-restore defects fixed alongside. The restore code is also inherited. RestoreTransitionedObject in the pools layer and the set layer, and updateRestoreMetadata, are identical to upstream master in all 18 SILO releases, and no SILO commit touches them.

The multi-pool deadlock came in September 2024, seven months after the expiry regression. The other three have been in restore since its first release in May 2021. The remaining restore repairs in this fix (version identity of the POST handler, encoded objects, and queued restores) depend on the same unchanged functions. They have not been traced to individual commits. The same holds for the older restore defects that the adversarial review found. The review history separates them from those the fix itself introduced.

Dates, PR numbers, and release membership come from the git history. The effects of the native-restore defects are those shown by the tests described under Verification. The statements about the empty version ID and about the origin of each native-restore defect come from reading the code.

Options

Option 1: pass the exact version (PR #282)

Always pass the version ID in the restored-copy branch, and map an empty ID to null. Then the current-version case takes the same path the noncurrent case already takes. This is the smallest change that reaches the root cause, and it is adopted.

Option 2: collapse the two lifecycle actions into one

Making the evaluator return DeleteRestoredVersionAction for every version would have the same effect at the call site. It would also change the action names seen by audit, tracing, and other callers of the lifecycle package, for no added safety. Rejected.

Option 3: a dedicated “drop restore cache” storage call

A separate storage operation would make it impossible to confuse cache cleanup with deletion. It would also add a new storage RPC, and rolling upgrades would have to handle nodes that do not know it. This is a reasonable long-term direction, not a release fix.

Option 4: guard only in the storage layer

A guard can refuse to remove data that is not a restore cache. It cannot stop the object-delete path, which writes a delete marker before any cache logic runs. The storage guard is necessary as a second line of defense but is not sufficient alone.

Proposed fix

The fix has five groups. The first is PR #282 as submitted. Groups 2 to 4 are maintainer follow-ups from the first two review rounds, and group 5 comes from the adversarial review.

1. Address the exact version (PR #282). expireTransitionedObject always passes the version ID in its restored-copy branch. An empty ID is sent as null, because the ListObjects max-keys=1 shortcut reads the latest version without the bucket’s versioning, so a null version can arrive with an empty ID even in a versioned bucket.
2. Keep the versioning state. The bucket’s suspended state is now set before the restored-copy branch returns, not after it. Without this, the cleanup of a restored noncurrent null version in a suspended bucket reached storage without that state; this layout adds eight cases to the contributor’s test, and all of them failed without the change.
3. Clean up only what is proven to be a restore cache.
   • The cleanup needs fresh metadata. If the set cannot establish read quorum, it does not fall back to the blind delete that ordinary unversioned DELETE may use.
   • If a queued cleanup for the null version now finds a replacement delete marker, it returns VersionNotFound instead of acting on it.
   • Read failures during restore or cleanup no longer trigger dangling-object removal of the version being rehydrated.
   • In xlMetaV2.DeleteVersion, a restored-copy cleanup checks the stored version before any other mutation. Only a stored object version whose transition is complete owns a restore cache; for anything else the call is a no-op. The check uses what is stored, not what the request claims. Legacy xl.json objects cannot hold a completed transition and are left untouched.
4. Make native restore safe to rely on. Once expiry is correct, restore becomes the remaining way to damage the version. Review found each of the following against main and reproduced it in tests, and each is fixed:
   • Version identity. The background restore keeps the bucket’s versioning context. Before this, restoring a null version could create a new version ID pointing at the same remote object. A later expiry of either version could then remove the shared remote object; that consequence is reasoned from the code path, not reproduced end to end.
   • First read failure. If the first read of the target version fails, restore returns without writing anything. Before this, requesting a missing version created a new, empty null version.
   • Multi-pool deadlock. A restore no longer runs inside an object lock that the pools layer already holds. It stages its copy without the object lock and takes the lock itself, briefly, for its commit and for its status writes. Before this, a multi-pool restore waited on its own lock until it timed out.
   • Version routing. The pool is chosen by the requested version, not by the latest version’s write location. Before this, restoring an old version in pool 0, while the latest version lived in pool 1, failed with VersionNotFound.
   • Encoded objects. Rehydration keeps the ETag, the original part numbers that encrypted multipart objects use to derive their keys, the logical sizes of compressed parts, and the part index. Objects written by older releases with legacy part sizes are handled.
   • Object Lock integrity. A restore no longer writes back a snapshot it took before the download. Its commit reads the version again under the object write lock and takes Object Lock, tags, and replication state from that revision, so a legal hold or a COMPLIANCE retention extension acknowledged during the download is kept. Before this, a single-pool restore held no lock, and the snapshot taken at the start overwrote such a change.
   • Queued restores. Each version has its own restore lock, which a restore takes before the object lock, and only the short commit takes the object lock. A restore of another version of the same key therefore does not wait for this one’s download. A commit that cannot take the object lock within its bounded wait clears its own marker and deletes its staged copy. Before this, a restore that gave up after one lock timeout left ongoing-request="true" on the version for good, so every later POST failed with RestoreAlreadyInProgress.
5. Give each native restore an owner (adversarial review). With groups 1 to 4 in place, the adversarial review still found three kinds of gap. A restore could be reached by other callers. A restore could act on a minority view of the version’s metadata. And an old attempt could act on a newer one. Each item below closes findings that it reproduced in tests.
   • Attempt identity. Each POST that starts a rehydration writes an attempt ID into the version’s internal metadata. With it, POST records the identity it resolved: version ID, ETag, size and modification time. The background worker, its staging and its commit all carry the attempt. If the attempt no longer owns the version, the worker stops without committing or resetting anything. A later POST takes over an ongoing restore only when it gets the version’s restore lock, which a running worker holds. Otherwise it waits up to two seconds for that lock, then answers 409 RestoreAlreadyInProgress when the drives show the version ongoing and a retryable error when they cannot tell, instead of waiting for the download.
   • Captured identity. The worker writes with the versioning state and the explicit version ID that POST resolved, and a null target stays null. Before this, enabling versioning while a restore was queued gave the restored copy a new version ID that pointed at the same remote object. A replacement null object under the same ID counts as a different target.
   • Recovery. A worker that fails before it publishes anything resets its own attempt to a retryable state, with bounded retries, and deletes its staged copy. A publication that reached some drives but not a write quorum is converged by the worker that made it, under the commit’s lock hold and then in bounded in-memory retries (Native Restore in Two Phases).
   • Staged commit (replaced the private upload). The first design kept the restored data in a private multipart upload, hidden from public multipart calls by a marker, and staged its completion under a commit lock. It stranded an ordinary upload whose completion failed after staging, and it held reads and writes of the key behind the download. The next release stages the copy in the tmp volume, outside the object lock, and publishes it drive by drive during a short lock hold; see Native Restore in Two Phases.
   • Authoritative revision. POST, the worker and the reset read the exact version from every drive. They use the metadata revision that at least max(n/2+1, read quorum) drives hold, identified as in the multi-pool fix: the version header plus a digest of the normalized metadata.
     • They change only the restore fields of that revision, and write it only to the drives that hold it. Before this, a status write could copy one stale drive’s metadata to every drive, and an acknowledged legal hold was lost.
     • If a write fails, the drives get back the revision they had.
     • When drives differ only in restore fields, the revision without those fields still identifies the version. A later POST, or the attempt’s own reset, can then bring the drives back into agreement. A reset applies only when its attempt is the only one in progress.
     • Without such a revision, the request fails and writes nothing.
   • Expiry and heal use the same view. The fresh recheck of restored-copy expiry needs its own reason: a restored-copy expiry action, a valid restore deadline that has passed, a completed transition, and no restore in progress. Before this, another lifecycle action that was due could remove a renewed cache. The recheck decides on that authoritative revision. It refuses when the drives disagree on the restore state. Heal copies restore fields to an outdated drive only when a quorum holds the same fields; otherwise it waits.
   • Bounded waits. POST’s first read takes no lock. Its lock waits are bounded.
   • Rolling upgrades. Older nodes do not follow the restore lock, the attempt fence, or the staged commit. The node that serves a POST now asks every peer through an internal capability RPC, which names native restore protocol 2. Until every peer answers that it speaks it, POST answers 503 ServiceUnavailable and writes nothing. A queued worker asks again before it starts, and clears its marker when a peer does not answer.
   • Lock domains. The adversarial review reproduced DELETE and CopyObject racing a restore in a distributed single-pool deployment whose erasure sets do not share hosts. That inherited split is closed by the single-pool lock-domain fix, which this group depends on, and is not repeated here.

Why it is sufficient and necessary

Sufficient. Every path that expires a restored copy ends in expireTransitionedObject, now with an exact version. These paths are the scanner, GET and HEAD, listings, decommission, and drive heal, and the set and storage layers then act only on that version. Three layers back each other up:

• the version ID chooses the target;
• fresh metadata confirms the target and its expiry date;
• the stored transition state proves the data is a cache.

A wrong target that is not a completed tiered version cannot lose its data. The native-restore repairs ensure that the version being cleaned up is the one that was restored, with its protection metadata intact.

Group 5 makes those repairs hold under concurrency and partial failure:

• One attempt owns a version at a time, and only that owner can commit the restore or reset it.
• The copy is staged where no public caller can reach it and is published only under the object lock, after the checks of the two-phase commit; the gate keeps restores off while any node does not speak that protocol.
• POST, the worker, expiry and heal all decide on the same authoritative revision. When there is none, each of them fails closed.

The commit protocol itself is argued in Native Restore in Two Phases.

Necessary.

• Without the exact version, the reported defect stays.
• Without the versioning-state change, suspended buckets keep a failing layout.
• Without the metadata and storage guards, a stale or misrouted cleanup can still remove the only data of a version that is not tiered.
• Without the native-restore repairs, fixing expiry would make restore the path to the same damage: new version aliases, empty versions, lost legal holds, or versions stuck as “in progress”.
• Without group 5, the adversarial review reproduced the following:
  • a stale drive’s metadata overwriting an acknowledged legal hold on every drive;
  • an old attempt resetting a newer one;
  • expiry or heal acting on a minority restore state.

These changes have to ship together. Unblocking the multi-pool restore deadlock before the cleanup is correct would make the damaging cleanup reachable on more deployments.

What the guards do not claim. The storage guard only prevents a non-tiered version from giving up its data. If a wrong target is itself a completed tiered version, the guard passes, and the worst case is the loss of that version’s restore cache, not of its data. Correct targeting remains the job of the version ID and the metadata recheck.

Cost and compatibility

• No public API, configuration, metric, or dependency change. One internal metadata key is added: the attempt on the version. One internal peer RPC answers which native restore protocol a node speaks. Older releases ignore the key.
• Restored copies of current versions are now removed by themselves. Restored copies that stayed because the old cleanup failed are removed on the next pass.
• No wait for the download. GET, HEAD, and writes of the key no longer wait for a restore’s download; only the short commit takes the object lock. #284 is fixed by the two-phase restore (Native Restore in Two Phases). The commit’s lock hold grows with the version’s metadata and the drive count, not with the download time.
• Behavior changes of POST ?restore.
  • On an upgraded node, it answers 503 ServiceUnavailable and writes nothing while any peer runs an older release or cannot be reached. Older nodes still serve their old restore path; pause new restore requests during the mixed-version window.
  • It fails without writing when the drives do not agree on the version’s metadata.
  • While a restore of that version runs, it answers 409 RestoreAlreadyInProgress within a few seconds.
  • Errors other than an invalid object state keep their usual S3 mapping instead of all becoming InvalidObjectState.
• Heal of a version waits while its drives disagree on restore fields and no quorum holds the same fields.
• Rollback restores the old behavior, including the defect. Before a rollback, follow the downgrade rules in Native Restore in Two Phases.

Residual limits

• Existing damage is not repaired. Delete markers created by the old cleanup stay. They are ordinary markers and can be removed with DELETE ?versionId=, after confirming from the ILM audit log or bucket notifications that the marker came from a restored-copy expiry. Data already removed from a null version, and hidden versions already deleted by noncurrent rules, cannot be recovered by this fix.
• Lock domains in large single-pool deployments. In a distributed single-pool deployment whose erasure sets do not share the same hosts, DELETE and CopyObject lock in a different lock domain from PUT, metadata updates, and restore. A DELETE or copy during a restore can then race with it. This inherited split is broader than restore. The single-pool lock-domain fix closes it, and this fix must not ship without that one.
• Rolling upgrades. The gate stops new restores on upgraded nodes only. A POST that an older node serves still uses the old path. Upgrade and downgrade rules are in Native Restore in Two Phases.
• Restart. The retries that reset a failed attempt run in memory. After a restart the version stays ongoing until a later POST takes it over.
• Disagreeing drives. Heal and expiry of a version wait while its drives disagree on restore fields and no quorum holds the same fields. A later POST, or the attempt’s own reset, brings the drives back into agreement once enough of them answer. If the drives also disagree on other metadata, so that no revision has a quorum even without the restore fields, neither restore nor heal changes the version. It then needs a later metadata write, such as a tag update, that rewrites it on every drive. This double-fault case is reasoned from the code, not tested.
• Inherited and unchanged. The sweep of temporary files can remove an in-flight part of any upload that streams for longer than the stale-upload age. Ordinary GET and HEAD can still return a stale drive’s metadata when the drives agree on modification time and ETag but not on other fields. Restore decisions no longer depend on that read.

Verification

Groups 1 to 4, on the integration head.

• TestExpireRestoredCopy covers eight layouts across four lifecycle paths (scanner, HEAD, listings, and the synchronous call made by decommission and drive heal), on one pool and two. That is 64 cases: the contributor’s 56 plus 8 for the suspended noncurrent null layout. They fail on main and pass with the fix.
• TestExpireRestoredRequiresFreshMetadata, TestExpireRestoredRenewedOrMissing, TestExpireRestoredStorageBoundary, TestExpireRestoredKeepsReplacementDeleteMarker, TestExpireRestoredMetadataReadKeepsMinorityVersion, and TestExpireRestoredStorageRejectsOtherTypes cover the guards.
• The TestNativeRestore* tests and TestMultipartCompletionLockAlias cover the native-restore repairs: pools, version routing, initial failure, POST version identity, Object Lock updates during a restore, queued restores, read safety, and encoded and legacy objects.
• The full ./cmd suite and the targeted race selection passed on the head reviewed in the maintainer rounds.

Group 5. The patch added 36 tests, on one pool and two where the layout matters. The 13 tests of the private upload and of commit staging were dropped with that design, together with TestNativeRestoreConcurrentCompletion of group 4, which completed the restore’s upload from a client; Native Restore in Two Phases lists the tests of the two-phase restore. The other 23 remain:

• Status writes and attempts:
  • TestRestoreStatusUsesQuorumGeneration and TestRestoreAttemptGenerationFencing;
  • TestRestoreGenerationSplitNeedsQuorum, TestRestorePOSTStatusWriteSplit, and TestRestorePOSTStatusWritePartialFailure;
  • TestRestorePOSTKeepsConcurrentProtection and TestRestorePOSTRacesCacheExpiry;
  • TestRestorePOSTRejectsReplacedTarget, TestRestoreKeepsCapturedNullIdentity, and TestRestoreReplacementNullObject;
  • TestRestoreSupersededAttempt, TestRestoreRecoversStrandedMarker, and TestRestoreRestartTakeover;
  • TestRestorePOSTOngoingBound, TestRestoreDuplicatePoolCacheStates, and TestRestorePOSTControls.
• Expiry and heal: TestExpireRestoredNeedsOwnReason, TestExpireRestoredUsesQuorumGeneration, and TestRestoreHealNeedsRestoreQuorum.
• Rolling upgrades: TestNativeRestoreCapabilityGrid uses a real test grid with an old-style peer. TestNativeRestoreCapabilityCache, TestNativeRestoreRollingUpgradeGate, and TestNativeRestoreCapabilityRevalidates cover caching, reconnects and cancellation.

The regressions from the design rounds fail on the previous integration head and pass with the patch. Each regression from the patch review fails with only its own fix disabled and passes with it. The controls pass both ways.

The restore, lifecycle, multipart, tiering, expiry, stale-upload and heal selection passes: 260 tests in ./cmd. The lifecycle and grid packages pass. The race-detector selection of 63 restore and cleanup tests reports no race. gofmt and go vet are clean.

The adversarial review’s own probes ran on the final patch, and 33 of 46 pass. The other 13 are accounted for:

• three need the lock-domain fix. On the integration branch they show the races can no longer happen: COPY no longer overwrites the restored version, DELETE waits for the restore, and a public abort cannot find the hidden upload;
• one hits the inherited temporary-part sweep;
• nine assert outcomes that the agreed design changes on purpose. Examples are a heal deferral treated as fatal, a race that can no longer start, cleanups that now wait for the lock, and a stale drive left for heal rather than rewritten.

Real-tier restore. On 2026-11-05 a second SILO server served as the remote tier of a versioned bucket with SSE-S3 (a static KMS key) and compression. A current version, a noncurrent version, a 74 MB multipart object and an incompressible object were transitioned and then restored with POST ?restore. With the fix all four restores completed, and with the tier stopped every restored version still read back byte for byte, so the data came from the restored copy. On 69fcf50cd the restore of the encrypted, compressed multipart object failed with Part size for 1 should be at least 5MB, and that object could not be read with the tier stopped; the other three restored. The expiry of a restored copy was not part of this run, because the shortest restore lasts one day; the tests above cover it.

Upgrade from the previous release. On the same day a four-server cluster on one host ran RELEASE.2026-10-26T00-00-00Z with a remote tier and transitioned objects. The servers were replaced with the fix two at a time, because a starting server waits until another server runs the same binary. While two old and two new servers served requests, PUT, GET, listing and DELETE through every server succeeded, and POST ?restore through each new server answered 503 ServiceUnavailable. After the last server was replaced, the restore completed, the restored object read back with the tier stopped, and all 20 objects written before the upgrade matched.

Still to run before merge: remote CI on the final head.

Lock domains across hosts, mixed-binary clusters across hosts, hour-long storage stalls and clock skew between nodes have not been exercised.

Integration. The fix sits on #278 in one integration branch for the next release, with the lock-domain, heal dry-run, #259, #285 and Console fixes stacked on it. Every pick applied without conflicts. On that head the full go test -tags kqueue,dev -p 4 ./cmd/... ./internal/... suite passes, as do the CI race-detector selections and a race selection that covers every new test. An adversarial review of the combination found no defects. It checked lock domains, cleanup paths and purge against restore with executed probes.

The passing local runs use the kqueue,dev tags. The untagged Ubuntu CI command, CGO_ENABLED=0 MINIO_API_REQUESTS_MAX=10000 go test ./cmd/ -count=1 -timeout 30m, remains pending on the final commit; the local tagged result does not replace that gate.

Review history

References

• Issue: #281
• Contributor PR: #282
• Companion fix: Multi-Pool Lifecycle Expiry (#278)
• #284, fixed by the two-phase restore: Native Restore in Two Phases
• Upstream multi-pool restore lock: minio/minio#20495