Multi-Pool Deletion: Preserve the Latest Copy on Failure

Decision, October 24, 2026: merge the corrected PR #257. Commit aafe4f8886 fixes the multi-pool deletion failures reported in issue #256 and closes the in-flight failure window after preflight checks. The core invariant is direct: finish cleaning up non-authoritative copies in other pools before deleting the authoritative copy that determines current reads.

Denis Babchuk reported the issue and contributed the initial patch. Code review reproduced the failures, identified a remaining execution-time failure window in bulk deletion, and extended the fix with two-stage execution, per-item response identity preservation, partial-deletion retry handling, and shared tier-object protection.

Background and Earlier Work

SILO uses capacity-weighted pool allocation. Overwriting an unversioned key can place the new value in pool B while an older physical copy remains in pool A. Reads resolve the latest logical object across pools, keeping the stale copy invisible until a subsequent delete encounters an error partway through execution.

Earlier multi-pool consistency fixes relate to this change as follows:

Those earlier changes did not cover ordinary unconditional deletion or bulk DeleteObjects. See Multi-Pool Object Consistency for the shared locking and migration design, and Overwrite Pool Affinity for the evaluation that retained the existing allocator.

The Failure: A Delete Can Expose an Overwritten Value

Consider this initial state:

Pool A: reports/latest.json = "old"     ModTime = T1
Pool B: reports/latest.json = "latest"  ModTime = T2, T2 > T1
GET:    "latest"

If both pools execute a delete in parallel, pool B can finish removing "latest" before pool A fails with an I/O error. A single DELETE then returns an error to the caller, but the latest acknowledged value is already gone and the next GET returns "old".

Issue #256 documented four symptoms:

Running the author’s 11 regression subtests showed 10 failures on baseline main and 11 passes on the initial PR. However, review found that bulk deletion could still lose the latest copy when a drive failed after the preflight check.

Why Preflight Was Not Enough

The initial patch checked whether affected erasure sets had enough online drives before starting bulk deletion. While that rejects known offline quorum loss early, online drive counts cannot guarantee that subsequent metadata reads, writes, or shard deletions will succeed.

Review tested a 45-case fault-injection matrix that started with all drives online, allowed the authoritative pool to finish deleting the latest copy, and then injected a failure inside DeleteVersions on the non-authoritative pool (covering simulated EIO, simulated storage disconnects, and real filesystem permission errors across every authoritative-pool position in two- and three-pool layouts, for unversioned, explicit null, and UUID-versioned objects):

• Baseline main (000e653038): Masked the per-item failure in all 45 cases.
• Initial PR (a30270d9ba): Reported the error, but lost the latest value in all 30 unversioned and explicit-null cases. The 15 UUID cases used identical payloads across copies of the same version and retained a copy while reporting the error.

Review also uncovered a separate response-identity bug: an early-error DeletedObject omitted ReplicationState, which the HTTP handler requires to reconstruct the original request entry’s identity. In a signed bulk request with two blocked replicated version purges, the response returned one error entry and one empty deleted entry. Preserving only the key and version ID was insufficient.

Per-Item Deletion Contract

Under the existing shared object write lock, the repair enforces the following per-item rules:

1. Resolve the addressed object or version across pools and identify its authoritative copy.
2. Complete cleanup in every non-authoritative pool (Stage 0) before attempting to delete the authoritative copy (Stage 1).
3. If non-authoritative cleanup fails, return an error for that exact request item and leave its authoritative copy untouched.
4. Allow independent items in the same batch to complete normally.

Key operational boundaries:

• Per-item protection rather than whole-batch rollback: If an object has copies in three pools, Stage 0 may remove one older copy before failing on the second older copy. Stage 1 is then skipped so the healthy authoritative copy remains intact, and unrelated items that succeeded in the same batch are not rolled back.
• In-flight errors on the final authoritative delete: Once Stage 0 succeeds across all non-authoritative pools, Stage 1 deletes the authoritative copy. If a disk failure occurs midway through Stage 1 itself, the caller should inspect the object state and retry after recovery. The two-stage ordering specifically prevents removing the authoritative copy when another pool’s cleanup has failed.
• Explicit version or marker deletion: Deleting an explicitly addressed historical version or delete marker legitimately makes the next older retained version current under standard S3 versioning rules.

The Implemented Design

Single-Object Deletion

Ordinary unversioned deletion retires older copies before the newest copy and stops at the first failure. Conditional deletion and explicit-version deletion continue using deleteObjectReconciled. If the selected unversioned object has completed a tier transition, it also routes through deleteObjectReconciled so local cleanup preserves shared remote references.

When creating a delete marker for a single object held in multiple pools, pool selection skips pools known to lack delete quorum and chooses a readable holding pool that passes the check.

Bulk Data Deletion in Two Stages

The new deleteObjectsReconciled reuses the existing namespace write locks and objectPoolInfos resolver without introducing a second locking domain.

Lookup concurrency is capped at 32 objects. Each lookup inspects all pools for the addressed version, orders candidates by ModTime, and breaks exact timestamp ties by the lowest pool index to match read selection. Unreadable metadata is treated as an error rather than evidence of absence.

Hold the shared write locks for the batch's unique keys
Resolve each item's authoritative pool

Stage 0: bulk cleanup in all non-authoritative pools
         record errors against the original items

Stage 1: delete the authoritative copies only for items
         whose Stage 0 cleanup succeeded

Process marker creation after data/version removal
Release the locks

Pools still execute in parallel within each stage using the existing erasure-set and disk-level bulk interfaces. Because two objects in the same batch can have their authoritative copies in different pools, a fixed global pool order cannot work; stages must be partitioned per item by authoritative pool.

Stage 0 also includes non-authoritative pools where lookup reported the object absent: a read-quorum absence does not prove that a temporarily unavailable drive holds no stale shard. If that pool cannot complete the delete, the item conservatively returns an error.

Every result entry initializes with the original key, version ID, and full ReplicationState, and explicit index mappings preserve request order across filtering and re-batching. The online-drive preflight remains in place to reject known quorum loss early, while two-stage execution enforces runtime safety.

Versioning and Marker Ordering

The batch entry point separates physical deletion from delete-marker creation using the same versioning rules as the lower layer:

Physical deletion runs before marker creation, preserving execution order when a single batch explicitly purges a null version and creates a new delete marker for the same key. Marker-only batches continue using their existing path (see Compatibility, Retry, and Operational Boundaries for degraded-mode marker edge cases).

Retry After Partial Disk Cleanup

A second failure mode arises when a pool deletes a version from some drives before failing on the remaining drives. On retry, drives that just deleted the version return success while drives that already deleted it on the first attempt return errFileNotFound or errFileVersionNotFound. Counting those two response groups separately can prevent either group from reaching quorum.

The fix in erasureObjects.DeleteObjects combines successful removals with errFileNotFound and errFileVersionNotFound only for pure physical deletion (unknown I/O errors never count toward quorum). Delete-marker creation and replication-state updates still require actual successful writes to reach quorum so an absent version cannot falsely acknowledge a marker or metadata transition that was never written.

Both single-version and bulk deletion paths (including marker writes on this path) use a delete quorum of floor(N/2) + 1, where N is the erasure set’s drive count.

Shared Tier Objects

Physical copies in different pools can reference the same remote tier object, including when one copy has been restored locally. Retiring a non-authoritative local record must not enqueue that remote tier object for garbage collection while the retained authoritative copy still references it.

Bulk items with completed tier transitions therefore route through the existing per-object reconciler, preserving SkipFreeVersion, replication state, and prefix versioning options. That subset executes sequentially, which also prevents concurrent metadata updates for duplicate entries in the same batch, while ordinary local objects stay on the two-stage bulk path. Added regression tests cover shared and distinct remote references, restored copies, pre-authoritative failure, and explicit SkipFreeVersion requests.

Verification at the Final Revision

The commits evaluated during review and verification are:

The committed fault tests start with all drives online, inject failures inside DeleteVersions, and assert both an accurate per-item error and zero deletion attempts against that item’s authoritative copy, verifying GET during failure, GET and version listing after recovery, retry completion, and normal completion of unrelated items:

The new regressions fail deterministically against the initial PR’s production code. The combined deletion, tier, and quorum test suite passed three consecutive -race runs before merge and an additional -race run after merge. Supplemental checks covered single-pool controls, prefix versioning configurations, directory and duplicate keys, Object Lock, version purges, and migration regressions.

Both final-head Go CI and merged-main Go CI passed full cmd and internal suites, race suites, static analysis, and cross-compilation for Linux, macOS, and Windows on amd64 and arm64, alongside DCO and VulnCheck.

Regression test command:

go test -race ./cmd \
  -run '^Test(Pools(Delete|BulkDelete|RetiringCopyPreservesSharedTierObject)|BulkDeleteQuorumEvidence)' \
  -count=1 -timeout=5m

Live mcli verification ran against a two-pool server process (four directory drives per pool): populate data on the baseline binary, restart onto the fixed binary to verify reads, bulk deletes, UUID version deletes, enabled/suspended delete markers, and new writes, and then restart the baseline binary over the same data directories.

Performance: Measured Deletion Cost

Benchmarks compare baseline 000e653038 against final PR head a2073d2514 (identical to merged main).

Method

• Hardware: Apple M5 Max (18 physical cores, 128 GiB RAM) with one 8 TB Apple SSD (APFS, ~95% full during testing; all logical directory drives shared this physical device).
• Configuration: Four directory drives per pool (EC 2+2), 4 KiB objects, 8 concurrent workers, GOMAXPROCS=8, exercising the object layer directly.
• Matrix: Five pools / copies layouts (where “copies” denotes physical copies of one logical object across pools rather than erasure shards), testing both single DELETE (1,024 requests per run) and 64-object bulk deletes (64 requests / 4,096 objects per run).
• Sampling: Three alternating pairs per layout and operation (baseline→fixed, fixed→baseline, baseline→fixed), recreating the identical initial state before timing each run. Total: 60 run samples, 30,720 single-delete requests, 1,920 bulk requests, and 153,600 logical objects deleted with zero errors.

Results and Variation

Throughput is measured in logical objects per second, showing the three-run median and observed range; paired change is the median of the three paired percentage changes.

Per-request latency in milliseconds (covering the full 64-object batch for bulk calls, shown as baseline→fixed):

P99 uses the nearest-rank method (with 64 requests per bulk run, P99 is close to the per-run maximum). Single-pool controls retained their scheduling path and showed no regression. Three-pool bulk throughput decreased by 15–17%, with a 22.5% P99 increase on the 3-pool / 3-copy layout; two-pool / two-copy single deletion decreased by 16.3% while bulk deletion in that layout stayed within run-to-run variation.

Where the Cost Comes From

Single deletion serializes cleanup of older copies before the newest copy. Bulk deletion adds cross-pool metadata resolution and orders Stage 0 before Stage 1 while preserving cross-pool parallelism and disk batching within each stage. For a batch of $B$ items across $P$ pools with $N$ drives per erasure set, resolution inspects $B \times P$ pool states and the online preflight checks up to $B \times P \times N$ drive states. Because the shared key write locks span this work, the added latency can delay concurrent writes to the same keys. Accepting this multi-pool deletion cost is necessary to guarantee that failed deletes never drop the latest copy.

Compatibility, Retry, and Operational Boundaries

• Error codes and client retries: A single-delete write-quorum failure typically returns HTTP 503, whereas bulk DeleteObjects can return HTTP 200 with per-item errors such as SlowDownWrite in the response XML. With MaxRetries=4, the tested S3 SDK did not automatically retry an HTTP 200 bulk response containing per-item SlowDownWrite, while mcli rm surfaces the per-item failure. Callers must inspect bulk response entries and retry failed items after storage recovery, preserving the target versionId where applicable.
• Concurrency and cancellation: The shared object write locks serialize multi-pool deletion against participating pooled writes. Because the HTTP bulk-delete handler uses a detached background context, a client disconnect does not cancel an in-flight batch.
• Suspended-bucket degraded bulk markers: Supplemental testing noted an existing edge case already present on baseline main: in two- and three-pool layouts, when a versioning-suspended bucket with UUID-only history receives a bulk delete marker during drive degradation, explicitly purging the null marker after drive recovery can still return a quorum error (one run exposed the retained UUID history and another kept the marker). The marker-only bulk path remains unchanged here.
• Object Lock and conditional delete: Object Lock and retention authorization checks are unchanged. Batch XML ETag conditions remain unsupported; see Conditional DELETE.
• Upgrade and rollback: No S3 parameter, RPC wire format, on-disk format, dependency, allocator rule, or configuration flag changes. During a mixed-version rolling upgrade, an unupgraded coordinating node still applies the older deletion path; rolling back to an older release cannot reconstruct a latest copy that was already lost.

Rejected Alternatives