• Design
• Lifecycle
• Batch Expiration
• Data Safety

This record covers a defect found during review of the #278 fix (Multi-Pool Lifecycle Expiry). It was evaluated against main at 69fcf50cd and against the maintainer integration branch for #278 and #281, which is not yet published.

Status: fix approved by review and committed locally. Tested together with the other fixes for the next release on one integration branch: the full test suite, the CI race selections and an adversarial review of how the fixes interact all pass. Not merged or released.

Short answer

The defect is real, and it blocks the next release. Three operations remove “all versions” of a key:

• the lifecycle extension ExpiredObjectAllVersions;
• the lifecycle action DelMarkerExpiration;
• batch expiration with purge.retainVersions: 0, which is the default.

All three do it with a recursive delete of the key’s directory on the drives. Object a/b is stored inside the directory of object a. Purging a therefore also removes a/b, a/b/c, and every other key that begins with a/. Those keys are removed in every erasure set and every pool, whatever their age, and whether or not any rule selects them. The purge writes one ILM audit entry and one bucket notification, both for a. Nothing records the other keys.

The same path has two further faults. It skips the recheck that should confirm the expiry under the object lock, so a version written after the scanner’s decision is deleted too. It also drops the metadata of tiered versions without leaving tier free versions, so their remote objects are never cleaned up.

The fix replaces the recursive delete with a purge of the key’s own versions by version ID. The purge runs under the object lock that the delete already holds and decides again on the versions it reads under that lock. It then deletes them one at a time, oldest first and the latest last, through the version delete of the drives, which removes the object directory only when it is empty. Keys with a transitioned version are refused in this release and keep everything. The change sits on top of the #278 fix, which touches the same function, and on the single-pool lock-domain fix.

Classification

When it happens

Three callers ask the pools layer for an all-versions delete:

• The expiry worker, applyExpiryOnNonTransitionedObjects. It runs expiries that the scanner queues, and also those that GET, HEAD, and listings queue when they see a due object.
• The migration expiry, applyMigrationExpiry. Since #276, decommission and rebalance call the worker’s function directly during the walk.
• Batch expiration, batchObjsForDelete, for rules whose purge retains no versions.

We probed each one on main. In every layout, the old key is due and every other key is new:

The probes give the same results on the #278 integration branch. That change does not mitigate this defect.

Root cause

The delete passes through three layers, and each one widens it.

1. The callers ask for a prefix delete. For Action.DeleteAll(), applyExpiryOnNonTransitionedObjects sets DeletePrefix and DeletePrefixObject. Batch expiration sets the same two flags.
2. The pools layer drops the other options. erasureServerPools.DeleteObject answers every DeletePrefix request with deletePrefix. That function calls each pool with a fresh ObjectOptions{DeletePrefix: true}, which loses two things:
   • Expiration.Expire. The recheck in erasureObjects.DeleteObject, which upstream added for exactly this action, therefore never runs.
   • DeletePrefixObject. erasureSets.DeleteObject therefore sends the delete to every erasure set of every pool, and it discards their errors.
3. The drives delete a directory. erasureObjects.deletePrefix calls Delete with Recursive and Immediate on every drive. On a drive, object a is the directory a/, which holds its xl.meta and data directories. Object a/b is a/b/xl.meta, inside that directory. The recursive delete moves the whole tree to the drive’s trash and purges it at once.

Directory objects escape only because a/ is stored as a__XLDIR__, next to a/ rather than inside it.

The same path has three side effects:

• No recheck. The scanner may have decided minutes or hours earlier. On one pool, a version written after the decision is deleted with the rest; we reproduced this. Review of the #278 fix found a related case, which we reasoned through but did not probe: a decision made on incomplete metadata, for example with half of the drives unreadable and one of the rest missing a new version, deletes a committed version.
• Tier leak. Deleting xl.meta wholesale removes the record of every transitioned version, including free versions that earlier deletes left for tier cleanup. Their remote objects stay in the tier, and nothing refers to them any more. We reproduced this for the key and for its children.
• Errors are ignored. The fan-out across erasure sets reports success even when a set’s delete fails.

Where the behavior came from

Provenance

The defect came from upstream MinIO. SILO inherited it at the fork and did not introduce it. The fork point is 27742d469 (2025-12-03). Every function on the defective path is unchanged from upstream master (7aac2a2c5, 2026-02-12): erasureServerPools.deletePrefix, erasureSets.deletePrefix, erasureSets.DeleteObject, erasureObjects.deletePrefix, and batchObjsForDelete compare equal to upstream, and so does the DeletePrefix branch of erasureServerPools.DeleteObject and erasureObjects.DeleteObject. The lifecycle caller keeps its upstream options for a delete-all.

The defect is the combination of four upstream ingredients. The first release in which a lifecycle rule could delete other keys is RELEASE.2023-06-29T05-12-28Z. The first SILO release with the defect is RELEASE.2025-12-03T12-00-00Z, the first SILO tag. All 18 SILO releases from there through RELEASE.2026-10-26T00-00-00Z contain it.

Upstream ingredients.

Upstream changes that touched the same functions and left the effect alone:

• 1a884cd8e (minio/minio#13242, 2021-09-17, RELEASE.2021-09-18T18-09-59Z) made the fan-out across erasure sets concurrent and made it discard errors.
• b49fc33cb (minio/minio#15148, 2022-07-11, RELEASE.2022-07-13T23-29-44Z) made the force delete purge the drives’ trash immediately.
• 88a89213f (minio/minio#19231, 2024-03-09, RELEASE.2024-03-10T02-53-48Z) made that purge non-blocking.
• 6a1558081 (minio/minio#19685, 2024-05-06, RELEASE.2024-05-07T06-41-25Z) made the per-set drive delete return write-quorum errors. The fan-out across sets still discards them.

Upstream changes that did not narrow the path. None of the three takes effect, because the pools layer rebuilds the options:

• eb55034df (minio/minio#17827, 2023-08-09, RELEASE.2023-08-09T23-30-22Z) added DeletePrefixObject. It is an optimization that avoids the fan-out for configuration deletes. It does not protect other keys.
• c61dd16a1 (minio/minio#19365, 2024-03-27, RELEASE.2024-03-30T09-41-56Z) set that flag in the scanner and in batch expiration, again “to avoid fan-out calls”.
• 289223b6d (minio/minio#19359, 2024-03-27, RELEASE.2024-03-30T09-41-56Z) added the recheck under Expiration.Expire in erasureObjects.DeleteObject. It is the only one of the three aimed at safety, and the pools-level call never reaches it.

SILO releases. The tags RELEASE.2025-12-03T12-00-00Z through RELEASE.2026-10-26T00-00-00Z (18 tags) all contain the pools-level and set-level prefix functions, the scanner and batch call sites, and the drive-level delete in their upstream form.

SILO commits that touched the path. None changed the prefix delete itself. Their effect on the defect:

So the defect is not a regression in any release. The only SILO change that makes it easier to reach is on main and not released yet.

Established from git history, not by running: the introducing commits, their dates and the first upstream release for each, the byte-for-byte equality of the functions listed above with upstream master, and the contents of each SILO release. The behavior of each ingredient commit as a release was not rebuilt and run.

Context. Upstream master still has the same code. In SILO, #276 makes the migration walks run this expiry immediately.

Options

Option A: an exact-object delete on the drive

A new drive operation would remove xl.meta and the data directories it references, then remove the object directory only if it is empty.

• It protects the children.
• It deletes every version the drive holds, including versions written after the decision or missing from the quorum view. It still needs the recheck, and it cannot limit itself to the decided versions without taking a version list. At that point it is the existing DeleteVersions call.
• To avoid the tier leak, the drive would have to write free versions itself, which duplicates what the version-delete path already does.
• It needs a new storage call. A new flag on the existing Delete call would be unsafe: older nodes decode DeleteOptions and skip unknown fields, so during a rolling upgrade they would run the recursive delete on their drives.

Rejected: more storage code, an upgrade hazard, and it still needs Option B’s logic.

Option B: purge the key’s own versions by ID

This is the recommended option.

• Children. The version-delete path (xlStorage.deleteVersions) rewrites or removes xl.meta and moves the data directory of each removed version to the trash. It removes the object directory only when that directory is empty, so other keys stay. A prototype through the existing bulk path deleted all four versions of a key, a delete marker and a tiered version among them, and left the children in both erasure sets intact.
• Tiered versions. xlMetaV2.DeleteVersion adds a free version for each transitioned version. The prototype left one, and the scanner’s free-version cleanup removes the remote object later (Tier Free Versions).
• Object Lock. Lifecycle rejects both rule types on lock-enabled buckets at configuration and skips them at evaluation; the purge does not change that. Because the purge deletes by version ID, it can enforce retention and legal hold per version in this one place, so the primitive defends itself whatever its caller.
• Replication. Delete-all stays a local purge. Versions are removed without replication state, no delete marker is created, and nothing is queued, as today.
• Quorum and partial history.
  • The purge deletes only the versions it decided on. A version missing from the quorum view stays, and heal reconciles it.
  • A write-quorum failure leaves the key in place and the next scan retries.
  • When a long list is split into several calls, the oldest versions go first and the latest version last, so a partial purge never changes the current version or exposes an older one.
• Multiple pools. The same version list goes to the key’s erasure set in every pool. A pool that does not hold a version answers “not found”, which the bulk path accepts for a physical delete. The fan-out to every erasure set goes away.
• Cost. Each drive in one set per pool does one read-modify-write of xl.meta and one data-directory rename per version, all under the object lock. Very long histories are split into several calls.
• Rolling upgrade. The purge uses the existing DeleteVersions call, with no new wire field or format change.
• The #278 check. The merged check already reads the versions in all pools under the lock and evaluates the rule. The purge can use that list instead of a yes-or-no answer.

The design review changed three points of this option; Proposed fix describes the result. The bulk path reorders the versions of a key and has no barrier between pools: in a two-pool probe it removed the latest delete marker while the delete of an older version failed, and a read then returned that older version. The purge therefore deletes one version at a time. A free version alone does not make tiered keys safe here: it hands the remote object to the tier cleanup, and the review showed cases where a copy in another pool, or a delete that missed its quorum, still referenced that object. Keys with transitioned versions are therefore refused. And because duplicate copies and null writes repeat a version ID, the purge records every copy in every pool before it deletes anything.

Option C: refuse when other keys exist under K/

A one-entry listing of K/ before the purge would skip such keys.

• It races: a key written under K/ after the listing is still deleted, because the lock on K does not serialize writes to K/child.
• The rule silently never applies to such keys.
• Every purge pays for a listing across all erasure sets.
• The tier leak and the missing recheck remain.

Rejected.

Option D: keep the prefix delete, but pass Expire and DeletePrefixObject through

This restores the recheck. With those options, the set-level delete refused a version written after the decision. It does not protect the children: a child in the key’s own erasure set is still deleted, while a child in another set survives. The loss would then depend on hashing, and the tier leak would remain.

Rejected as the fix. Its recheck is part of Option B.

Proposed fix

This is the design agreed in two design review rounds and implemented in the reviewed patch.

1. A dedicated option. The expiry worker, the migration expiry, and batch expiration request the purge through a new in-memory option, ObjectOptions.DeleteAllVersions, and stop setting DeletePrefix and DeletePrefixObject. DeletePrefix keeps its meaning for force delete and for internal callers that mean a prefix. The option is rejected together with DeletePrefix, a version ID, data movement, or any option that creates a delete marker. Every layer below the pools layer rejects it too, so a misrouted request never turns into another kind of delete.
2. Decide on the history under the lock. erasureServerPools.DeleteObject hands the request to the purge right after it takes the object lock, before the #278 check, on single-pool deployments too.
   • Lifecycle: the unchanged lifecycleEventsAllPools evaluates the rule on the versions of the key in all pools. The purge runs only if the latest version still carries a delete-all event.
   • Batch expiration has no rule to evaluate. The latest version must still be the one the walk matched, by version ID (an empty ID counts as null) and modification time. Otherwise the key is reported as not expired and is not retried.
   • A second read of the authoritative history of each pool records where each approved version lives: every copy, including older writes of the same ID such as null versions written in several pools. This read may only confirm the approval; it never adds a version. Any difference, or a history that the drives cannot tell, returns InsufficientWriteQuorum and deletes nothing. A key without versions is a success that removes nothing.
3. Check every write before deleting any.
   • The purge reads xl.meta on every drive of each set that holds a copy. A minority of drives can hold another revision of a version than the one the history accepted, so every revision of an approved version on every drive is checked. A drive that cannot be read could hide such a revision, and heal would later remove what a purge left on it, so an unreadable drive refuses the whole key. The purge then deletes on exactly the drive handles it read; a handle that a reconnect installs later is never used.
   • If any revision of any version is transitioned, the whole key is refused. See Tiered keys.
   • The purge enforces retention and legal hold on every write it would delete. A protected revision refuses the whole key.
4. Delete in order, by version ID.
   • One version at a time, oldest first, through the version delete of the drives (DeleteVersion) in the key’s erasure set of each pool that holds a copy, on the drives it checked. That call rewrites xl.meta, moves the version’s data directory to the trash, and removes the object directory only when it is empty. The delete writes no marker, no replication state, and no free version.
   • For each version, older writes go before the copy that reads choose.
   • The latest version goes last and alone, only after every older version is gone from every pool. A purge that stops therefore never changes what a read returns, and never exposes an older version.
   • The purge stops at the first error. A delete that reached its write quorum but failed on some drives is repeated on those drives a bounded number of times while the lock is still held; what remains is left to heal.
   • Each call has a budget of 1000 versions and one minute, its reads and checks included, checked before each version. A longer history stops before the latest version and continues on the next expiry, with a fresh history and a fresh decision.
5. Report what was removed. The purge counts the logical versions it removed, delete markers included, duplicate copies and free versions not, and returns the approved latest version. The ILM audit entry carries the count in the tag ilm-versions-removed. A purge that fails or stops after it began to delete is audited as partial (tag ilm-partial) with the count and the error, also when it removed copies but no whole version, and it sends no removal notification. A purge that removed nothing sends nothing. Batch expiration counts objects and delete markers from the purge’s results, and a job that failed to remove a delete marker is reported as failed.

Tiered keys

Delete-all of a key with transitioned versions does not change in this release.

• The expiry worker still sends a delete-all of a transitioned latest version to expireTransitionedObject. That writes a delete marker and keeps every version and remote object under it, and no later scan removes them.
• A key whose latest version is hot but whose older versions are tiered is refused. Nothing is deleted, and the scanner tries again on each pass. Batch expiration reports the key as failed and does not retry it within the run. During a decommission or rebalance the key stays in its source pool, and the decommission reports it as left behind, as it already does for a transitioned latest version (executed).

A tier-safe purge must not hand a remote object to the tier cleanup while any copy still references it, including after a delete that failed its quorum. That needs durable cleanup records and proof under the lock, and it is a separate change.

Coordination

• The #278 fix. This change sits on the #278 commit and uses its helpers unchanged: lifecycleEventsAllPools for the decision and objectVersions for the per-pool histories. The purge runs before the merged check, so it needs neither CheckedAllPools nor the single-pool exemption.
• The single-pool lock-domain fix. On a single pool, writes of a key lock in its own erasure set, while DELETE locks in the first set; see Lock Domains of Large Single-Pool Clusters. The purge decides and deletes under the DELETE lock, so it excludes every writer of the key only once that fix is in. It lands first, and this change is rebased onto it. The purge adds no locking of its own.

Why it is sufficient and necessary

Sufficient. Every delete-all reaches erasureServerPools.DeleteObject. That includes the expiry worker with all its producers (scanner, GET, HEAD, listings), the migration expiry, and batch expiration. With the dedicated option, none of them reaches a recursive delete:

• A version delete touches only the key’s own xl.meta and data directories. Children have their own xl.meta, and the object directory is removed only when it is empty.
• The recheck runs under the lock that serializes writes to the key, with the same evaluator as the scanner, so a decision that no longer holds deletes nothing.
• Deleting only the approved list keeps versions that the quorum view missed.
• Deleting the latest version last keeps what reads return until every older version is gone.
• Refusing keys with transitioned versions keeps every remote object referenced.

Necessary. Each part closes a path that otherwise stays open:

• Without the exact purge, children are deleted. Option D shows that even the single-set prefix delete loses them.
• Without the recheck on single-pool deployments, versions written after the decision are deleted.
• Without restricting the delete to the approved list, versions missing from the quorum view are deleted.
• Without the order and the barrier before the latest version, a failed delete in one pool exposes an older version.
• Without the tier refusal, remote objects leak, or the tier cleanup removes one that a copy still uses.
• Without the batch and migration routing, those callers keep the prefix path.
• Without a dedicated option, any caller that sets the old flags reaches the prefix path again. DeletePrefixObject has had no effect on this path for years without anyone noticing.

Deliberately excluded.

• Force delete and the speedtest cleanup keep the prefix delete; it is their contract.
• Internal configuration deletes (deleteConfig, delServerConfigHistory) take the same recursive path on the system bucket. A collision needs a configuration name that extends another record’s file name with /, such as an IAM user whose name begins with another user’s name followed by /identity.json/. We confirmed the mechanism at the configuration layer. We did not check whether the admin API accepts such a name. This is low severity, and a follow-up can move these deletes to the same exact purge.

Cost and compatibility

• No API, configuration, storage-format, or wire change. The new options exist in memory only.
• Per delete-all, the purge reads xl.meta of the key’s set in each pool three times: for the decision, for the placement, and to check every drive. Then, for each version, every drive of each set that holds a copy does one read-modify-write of xl.meta and one data-directory move. Today’s delete is one recursive delete on every drive of every set. Large clusters do less work; keys with long histories do more and hold the object lock longer, at most for the budget of one call.
• Trash is purged in the background instead of immediately, so space comes back slightly later.
• A delete-all whose decision no longer holds now does nothing, single pool included; the scanner retries it on a later pass.
• Rolling upgrade is safe, because only existing storage calls are used. Protection is complete only once every node that executes expiries runs the fix: the node that runs an expiry decides how it deletes, so an old node in a mixed cluster still runs the recursive delete. Keep the interim mitigation until every node runs the fix.
• Rollback restores the old behavior, and neither direction needs a migration.

Residual limits

• Past losses are not repaired. The recursive delete purges the trash immediately, so there is no practical on-drive undo.
• Orphaned remote objects stay. Remote objects already orphaned in a tier remain there. Their mapping to keys was in the deleted metadata, so cleaning them up is manual.
• Tiered keys are not purged. See Tiered keys.
• A delete-all waits for every drive. While a drive of the key’s erasure set is offline or cannot read the key, the purge refuses the key and the scanner tries again on a later pass. This is the price of checking every revision before deleting any.
• A drive handle names a drive, not a backend. The purge deletes on the drive handles it checked, but each storage call resolves the backend again on the node that serves the drive. A backend that replaces a drive while a purge runs starts empty, and it receives the key only from a write or a heal of the key, which take the object lock the purge holds. An admin heal started with nolock skips that lock; that option is a separate issue.
• Keys under an object key are not listed. A listing does not descend into the directory of an object key, so while K exists, K/child is missing from listings, from batch expiration walks, and from the walks of decommission and rebalance. This is older than the fix. With the fix, K/child survives a delete-all of K that a decommission or rebalance runs, but the walk has already passed it. The first pass reports it as left behind, as it does without any lifecycle rule (executed); a later pass moves it.
• Data directories are removed recursively. Every version delete, not only delete-all, moves the version’s data directory to the trash with its contents. A key named K/<data-dir>/..., where <data-dir> is the internal data-directory ID of a version of K, is removed with that version. The ID is not returned by the S3 API, so such a name needs internal metadata to construct. It is a separate storage-layer follow-up.
• Legacy-format objects are not covered. On drives from the oldest erasure format, objects still stored as xl.json are deleted recursively by every version delete, not only by delete-all. We reasoned this from the code and did not probe it.
• The batch recheck is narrow. Batch expiration can only confirm that the latest version is still the one it matched.
• Mixed versions. See Cost and compatibility: an old node that runs an expiry still deletes recursively.

Operator guidance until the fix ships

Find exposure.

• Lifecycle: run mc ilm rule export ALIAS/BUCKET and look for enabled rules that contain "ExpiredObjectAllVersions": true or DelMarkerExpiration.

• Batch expiration: run mc batch list ALIAS --type expire and mc batch describe ALIAS JOB-ID. Look for rules with no purge.retainVersions or with retainVersions: 0.

• Keys: in those buckets (or under the rule’s prefix), list the keys including versions and delete markers, then print each key that is a path prefix of another key:

  mc ls --recursive --versions --json ALIAS/BUCKET | jq -r '.key' | sort -u > keys.txt
  awk 'NR == FNR { k[$0]; next }
       { n = split($0, p, "/"); a = p[1]
         for (i = 2; i <= n; i++) { if (a in k) print a " -> " $0; a = a "/" p[i] } }' keys.txt keys.txt

  A line a -> a/b means that a delete-all of a also removes a/b. A right side that ends in / is a directory object, which is not at risk.

Mitigate until every node runs a fixed release.

• Disable the affected rules with mc ilm rule edit --id RULE-ID --disable ALIAS/BUCKET, or remove the element from the rule.

• Where the policy is still needed, use per-version rules:

  • in versioned buckets, Expiration Days together with NoncurrentVersionExpiration NoncurrentDays and ExpiredObjectDeleteMarker;
  • in unversioned buckets, Expiration Days alone.

  These delete by version and do not take the recursive path (reasoned from code). They take longer to remove everything. On multi-pool deployments they are exposed to #278 until that fix ships.

• A prefix or tag filter helps only if it excludes the parent keys. A filter that still selects a still removes a/b, because the children do not need to match.

• Do not run batch expiration jobs with the default purge on buckets that have such keys, and cancel running ones with mc batch cancel. retainVersions of 1 or more deletes by version (reasoned), but keeps that many versions.

Detect past losses.

• With audit logging, look for entries with API name ILMExpiry and tag ilm-action set to DeleteAllVersionsAction or DelMarkerDeleteAllVersionsAction. Bucket notifications s3:ObjectRemoved:DeleteAllVersions and s3:LifecycleDelMarkerExpiration:Delete carry the same information. Each one names only the parent key and the rule. There is no audit entry, notification, or trace for the keys removed with it.
• For batch expiration, the job status and the batch trace count and list only the keys the rules matched.
• For each parent key K found this way, compare the keys under K/ before the event with what exists now. Earlier listings can come from an inventory, a backup, the application’s own records, or a replication target. A key that is missing and has no delete record of its own was most likely removed by this defect.
• Delete-all is not replicated, so bucket replication targets may still hold those keys, unless the target runs the same rule. Site replication peers share the lifecycle configuration and apply it themselves, so they may have lost the same keys (reasoned).

Verification

The fix, executed. Focused go test -run selections with -p 2, on the fix and on a disposable worktree at 9f7b49369 with only the new tests added. Each regression below fails on that base and passes with the fix. The new tests pass with the race detector.

• Children in the key’s own erasure set, in the other set, and deeper under the key keep all their versions after a lifecycle delete-all, for ExpiredObjectAllVersions and for DelMarkerExpiration. The same holds for a child in another pool, with copies of the key in both pools, and for a batch expiration purge, whose counts come from the purge.
• Decommission and rebalance remove the due key and keep its new child.
• A key named ea/dir__XLDIR__/odd inside the directory of the directory object ea/dir/ survives the purge of ea/dir/.
• A version written after the scanner’s decision survives on a single pool. A batch purge whose matched version is no longer the latest deletes nothing and counts nothing.
• On a single pool, an expiry on a history that half of the drives cannot read deletes nothing.
• A null version written in two pools whose newer write cannot be deleted is audited as partial with no version removed, since a copy is gone.
• A batch purge that removes the older version but fails on the latest delete marker counts the marker as failed, and the job as failed.
• Two pools, three versions: when the delete of the middle version fails, the oldest is removed, the latest stays and is still what reads return, the audit trace is marked partial with one version removed, and no removal notification is sent. Once the fault is gone, the next expiry completes and sends one notification for two versions.
• A budget of two versions stops a four-version purge before the latest version, reports partial progress, and the next expiry completes it.
• Keys with a tiered older version under a hot latest version, and keys with a tiered latest version in a batch purge, are refused with every version and remote object kept; batch expiration counts them as failed.
• A write protected by retention or legal hold refuses the whole key, including a protected older null write in another pool.
• A transitioned revision, or a revision under legal hold, on one drive of sixteen refuses the whole key, although the accepted revision is neither. So does a legal-hold revision on a drive that cannot be read, and one unreadable drive with no protection.
• A drive handle that replaces a checked one after the check receives no delete.
• A budget of 10 ms with slow metadata reads stops the purge before its first delete.
• A null version written in two pools: when the newer write cannot be deleted, the older write is gone and reads still return the newer one. The two copies of one version in two pools count as one removed version.
• A free version left by an earlier delete of a tiered version stays.
• The option is rejected together with every conflicting option, and by the layers below the pools layer. An absent key is a success that removes nothing.
• Controls that pass on both: a delete-all of a key without children removes every version; ea/plainx and ea/plain-1 survive; the directory object ea/dir/ is removed and ea/dir/kid stays; force delete still removes the prefix; a transitioned latest version still gets a delete marker; on two pools a stale decision is still refused.
• The existing lifecycle, expiry, batch, ILM, delete, decommission, and rebalance tests in ./cmd, and ./internal/bucket/lifecycle, pass with the fix.

The defect, executed. These runs used a disposable worktree at main 69fcf50cd and focused go test -run selections with -p 2, without the race detector.

• Direct call. DeleteObject with the scanner’s options for a delete-all of eav-parent deletes eav-parent/child and keeps eav-parentx.
• Scanner and expiry worker.
  • Lifecycle applied through scannerItem.applyActions and the real expiry worker gives the results in the table above: one pool with two erasure sets, DelMarkerExpiration, an unversioned bucket, and a directory object.
  • The drive scanner’s own folder walk (NSScanner) queues the delete-all for the parent, and the worker then deletes its children.
• Other callers.
  • Two pools: a child in the second pool is deleted.
  • A batch expiration job started through BatchJobExpire.Start deletes the child of the key its rule names.
  • Decommission and rebalance remove both the due key and its new child from both pools.
• Side effects.
  • Tiered versions of the key and of its child leave their remote objects behind, with no metadata and no free version. A free version left by an earlier delete is wiped as well.
  • A version written after the scanner’s decision is deleted on one pool. Calling the set-level delete directly with Expire and DeletePrefixObject refuses it.
  • The ILM trace shows one expiry entry, for the parent only.
  • The object layer accepts ea/dir__XLDIR__/odd, and it is deleted with the directory object ea/dir/.
  • Saving two configuration records, one under the other’s path, and deleting the outer one also deletes the inner one.
• Transitioned latest version. A delete-all writes a delete marker, keeps both versions and the remote object, and the next scan queues nothing.
• Option D. The set-level prefix delete with the restored options deletes a child in the key’s own set and keeps a child in the other set.
• Option B prototype. Deleting the key’s versions by ID through the existing bulk path keeps all children and leaves a free version for the tiered version.
• The #278 integration branch. The single-pool, two-pool, migration, and single-pool stale-decision probes lose the same keys. The two-pool stale decision is refused there.

Reasoned from code, not executed:

• that the audit entry and the notification of the defective path are written only for the parent; they share the call site that produced the single trace entry;
• that GET, HEAD, and listings queue delete-all through the same worker;
• that the fan-out across erasure sets discards their errors;
• that older nodes skip unknown DeleteOptions fields;
• that per-version lifecycle rules and batch purges with retainVersions of 1 or more do not reach the recursive path;
• that replication targets keep the removed keys;
• the incomplete-metadata variant found in the #278 review;
• the legacy xl.json behavior;
• the immediate trash purge.

Not run:

• an S3-level reproduction against a running server;
• a multi-node cluster or a mixed-version upgrade;
• the full ./cmd suite and remote CI;
• the fix together with the single-pool lock-domain fix.

Integration. The fix was cherry-picked onto one integration branch for the next release, on top of #278 and #281 and together with the lock-domain, heal dry-run, #259, #285 and Console fixes. Every pick applied without conflicts. On that head the full go test ./cmd/... ./internal/... suite passes, as do the CI race-detector selections and a race selection that covers every new test. An adversarial review of the combination found no defects. It checked lock domains, cleanup paths and purge against restore with executed probes.

Review history

References

• Evaluated base: 69fcf50cd
• Related fix and record: #278, Multi-Pool Lifecycle Expiry
• Migration expiry: #276, ILM Expiry and Pool Migration Ordering
• Tier cleanup: Tier Free Versions Are Internal Cleanup Records
• Upstream history: minio/minio#12504, minio/minio#17521, minio/minio#17827, minio/minio#17946, minio/minio#19359, minio/minio#19365, minio/minio#19481
• Operations: Object Lifecycle Management, Batch Expiration, mc ilm rule add, mc ilm rule edit, mc ilm rule export