Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation

This page records the analysis, design choices, review, and implementation of #153, #152, and #137. They belong to the same replication reliability series, but affect operation classification, recovery visibility, and task lifecycle respectively. One general retry patch cannot repair all three.

As of 2026-09-09: PR #162 is merged as d1105bbb, and all three issues are closed. All eight checks on the tested PR head, followed by main Go CI and VulnCheck, passed.
Review: the plan was discussed with the installed Claude Code Fable 5.1 Max, followed by a review of the implementation. The final verdict was GO.
Delivery boundary: this work completed code, tests, and main integration. It did not create a Server tag or formal release, and does not establish that existing packages, images, or production deployments contain the fixes.

Overall decision and the surrounding series

The selection rule was to repair a reproduced invariant at the smallest boundary that owns it, retain existing recovery mechanisms, and use deterministic tests to prove that work can finish. Additional complexity needs a concrete counterexample.

The surrounding series had already separated three concepts that are easy to confuse:

• #136 / PR #138 repaired counter completeness: receive and apply the final result before persisting terminal state, without waiting for the one-minute periodic flush.
• #139 repaired outcome accuracy: an existing destination object does not prove that this update succeeded. Success and failure must come from the target’s actual replication result.
• #137 repairs cancellation and resource lifecycle: the task must stop, its walker, workers, and result consumer must exit, and an old run must not pollute a new run’s state. This change preserves the first two contracts rather than introducing another accounting mechanism.

Authorization to use internal replication semantics belongs to the earlier CORS and replication trust record. Authoritative Object Lock state across pools remains tracked separately in #133, still open at this record’s date. Closing these three issues does not mean every replication concern is resolved.

The release-gating target is the maintained pgsty/silo stack with Console, mcli, and silo-pkg. Compatibility with upstream MinIO/MC remains best effort. Neither a mechanism proposed for upstream nor an external experiment can automatically be treated as an observation on current SILO.

#153: distinguish marker replication from permanent version deletion

The report and the reproduction differ

The original issue described a sustained HTTP 405 storm involving ILM and replication, and proposed treating every delete-marker 405 probe as completion. Current SILO source and measurements do not justify adopting that explanation and patch directly.

The baseline was 450dcb848. The experiment used two locally built SILO servers, separate disposable data directories, and maintained mcli / minio-go clients. A critical observation was that mcli uses bulk DELETE even for a single key; that route was already correct. A direct single-object S3 DELETE was therefore necessary to exercise the faulty entry point.

Checking only that the remote version disappeared would falsely declare success. Source metadata must be inspected too. The reproduction establishes incorrect initial purge classification and completion state, plus an unnecessary probe. It did not reproduce the external report’s sustained 405 storm or request-volume figures. Existing scanner/heal and resync producers already select the proper purge path; they cannot be described as necessarily repeating that erroneous probe every cycle.

Why one classification condition is sufficient

When deleting an existing marker, the object layer can return both DeleteMarker=true and a nonempty VersionPurgeStatus. The former describes the version being operated on; the latter describes the operation now required. They are compatible facts.

The old single-delete handler checked only DeleteMarker and populated DeleteMarkerVersionID. Completion then updated ordinary ReplicationStatus instead of completing the purge. The final condition is:

if objInfo.DeleteMarker && objInfo.VersionPurgeStatus.Empty() {
    dmVersionID = objInfo.VersionID
} else {
    versionID = objInfo.VersionID
}

This matches the classification already used by other producers. The fix belongs in the DeleteObject handler. It requires no storage-format change, relaxation of replica deletion protection, or new recovery task. Legacy PENDING entries remain recoverable through the existing scanner/heal purge path.

Why a 405 is not unconditional success

A 405 from a versioned HEAD can establish that the marker exists at the target. For replicating a delete marker, that can mean idempotent completion. For permanently deleting that version, it means there is still work to do.

Writing VersionPurgeComplete merely because HEAD returned 405 could let the source remove its metadata while leaving the unwanted target version behind. The implementation retains existing 405 semantics. Real remote failures such as 403, 405, and 503 must not be disguised as successful permanent deletion.

Source history traces marker-based classification to an upstream 2020-11-19 commit. The 2023-07-10 optimization changed scheduling from dsc.ReplicateAny() to the returned object’s replication/PENDING purge state while retaining classification based only on DeleteMarker. At this location, the 2025-04-02 commit merely moved Pending to replication.VersionPurgePending; it did not first introduce that scheduling condition. This identifies lineage, not a bisect across every historical release, and does not establish when the entire externally reported storm was introduced.

#152: expose drops from an already bounded queue

MRF, or Most Recent Failures, records recent failed replication work for background processing. It already had limits: mrfSaveCh has capacity 100000, and mrfRetryLimit is 3. The code drops a queue entry when RetryCount > mrfRetryLimit or the save channel is full.

The source object and its pending replication state remain. Dropping a queue entry does not mean losing source data. However, subsequent repair depends on the scanner; a prompt retry can turn into a wait for a scan, so this mechanism does not establish a fixed recovery deadline.

The actual defect was that TotalDroppedCount / TotalDroppedBytes increased internally but were omitted from both public statistics snapshots and from Prometheus v2/v3. A deterministic capacity-one fixture demonstrates the failure: one entry is admitted, a 20-byte overflow entry and a 30-byte retry-exhausted entry are dropped, and internal totals become 2 / 50 while administration reports 0 / 0.

The selected minimum change

1. Atomically read the existing counters into both administration snapshots.
2. Register and load cumulative counters in metrics v2 and v3, and document their meaning.
3. Warn at the two actual drop sites: retry exhaustion and a full MRF channel. Fixed messages and deduplication keys prevent changing counter values from defeating deduplication. Handing ordinary worker overflow to MRF is not itself a drop, so it does not gain a warning here.
4. Route ordinary objects, healing, and deletion consistently by (bucket, objectName) to restore worker affinity for the same object.

These counters accumulate since Server startup and reset on restart. Operations count entries, not unique objects; one object can contribute repeatedly. Bytes cover known sizes, with deletion entries contributing zero bytes. They measure neither data loss nor the complete backlog. Operators should examine increases alongside replication backlog and target health.

This work does not enlarge the queue, increase retry limits, add a persistent retry scheduler, or introduce another backoff framework. Existing limits continue to bound memory use, and the scanner remains the eventual repair mechanism. The repair makes an invisible condition observable; it does not promise a recovery deadline under arbitrary failures.

#137: make cancellation part of a run’s lifecycle

One token cannot cancel a group

Previously, cancellation placed one unkeyed token into a shared channel. One site resync can cover several buckets, with up to 10 concurrent bucket runs, while dispatchers and workers compete to consume the token. Only one bucket might stop; an unrelated task might consume it; or a leftover token might affect a later task.

Two blocking phases had independent defects: a bare receive from Walk output did not observe cancellation, and sending to a full worker channel did not observe it either. Once a worker exited, the dispatcher could block forever on a channel with no receiver. Walk also inherited the parent context, so an early function return could not stop its own walker.

State handling had related faults: site updateState modified a local value without writing it back to the map; bucket Canceled handling was incomplete; and an old finalizer could overwrite canceled state with Completed.

Registration, cancellation, and status share a write boundary

Each resyncBucket creates an owned context.WithCancelCause and registers before waiting for a concurrency slot. Registration and cancelResyncID share the resyncer’s state lock:

• Registration validates that the target still exists and the resync ID still matches, and reads its current cancellation state.
• Cancellation marks matching Pending / Started states Canceled before canceling all matching registered contexts.
• A registered run still waiting for a slot receives cancellation; a run registered after cancellation sees the canceled state.
• Walk, dispatch, workers, and result sends observe that run’s context. Unrelated IDs are unaffected, and there is no buffered token for a later task to consume.

A dedicated operation mutex serializes site start/cancel configuration setup. Canceling running work still executes if part of the target-configuration loop fails. Bucket finalizers and counter updates check the current target and resync ID, ignoring late results from removed, replaced, or canceled runs. Site state is actually written back, and a late Completed result cannot overwrite Canceled.

Success and failure require different finalization order

The result-consumer contract from #136 must remain intact: persist terminal state only after the consumer finishes.

Success: close worker inputs → join workers → drain results
         → persist terminal state → return slot → cancel owned context and unregister

Failure/cancellation: cancel context first to unblock work → join workers and consumer
                      → persist the appropriate terminal state → return slot → unregister

Canceling before joining workers on a normal completion path could discard pending work and turn a successful run into Failed. The final code calls finish exactly once from one defer, using defer order to complete cleanup. It therefore needs no additional sync.Once.

WithCancelCause distinguishes an explicit user cancellation from parent-context interruption. User cancellation becomes Canceled; parent interruption observed during finalization must not leave an interrupted run marked Completed. Shutdown while a run is still waiting for a slot preserves the existing Pending state for restart recovery.

Protect terminal state and resumed work

Cancellation can still arrive between the context check and the terminal save. Under its lock, markStatus must persist an existing Canceled state even if the finalizer previously computed Completed. An old resync ID must also be unable to overwrite a new run.

This is not a transaction across metadata files. If a bucket completed and persisted before cancellation, a subsequent site cancellation can leave different records reading “bucket Completed, site Canceled.” Completion happened first. The guarantee is that late completion cannot turn an already canceled run back into Completed, not that cancellation erases work completed before it.

The recovery loader, loadResync, previously launched goroutines and then immediately executed defer cancel(). Inspection of SILO’s actual shared-lock implementation confirmed that this cancel ends the merged leader context; it is not a no-op. A WaitGroup now retains that context until resumed runs exit. Losing leadership still cancels the existing context; replacing it with a global context would bypass the leadership constraint. Loading disk state must also preserve newer in-memory start/cancel state.

Fable review and the complexity decisions

The review used the installed Claude Code with model argument claude-fable-5-1[1m] and --effort max. Baseline reproductions and the minimal proposal produced agreement on all three repairs. The final patch and validation were then reviewed again, yielding GO. The conclusion rests on code and evidence, not model agreement alone.

The final review accepted two further implementation boundaries. The infrequent resync start operation holds the status lock across configuration reads and writes, consistent with existing terminal saves. The active registry uses resyncOpts, including resyncBefore, as its key; current callers reuse the same in-memory values, and no identity mismatch was reproduced. Future changes that reconstruct time values or restore task identity should revisit equivalence, rather than adding another registry without evidence now.

Validation and reproducible evidence

Regressions were added against unchanged production code first, and failed before the fixes. New tests exercise the production handler, real erasure storage, actual metric registration, and resyncBucket, rather than only testing helper logic that repeats the implementation.

Tests pinned to the merged revision: delete markers, MRF visibility, and cancellation lifecycle. With that revision and Go 1.27.1, the relevant checks can be repeated with:

go test ./cmd -run '^(TestReplication|TestReplicateDeleteMarker|TestResync|TestSiteResync)' -count=1
go test -race ./cmd -run '^(TestReplication|TestReplicateDeleteMarker|TestResync|TestSiteResync)' -count=1
go test ./cmd -run '^TestResyncCancel' -count=100
go test ./... -count=1 -timeout=30m

After complete local validation, only new-test formatting and fixtures changed: an existing ARN was reused, and the collector supplied the metric prefix, preventing the compatibility scanner from treating test strings as new protocol identifiers. The guard was not weakened. Relevant tests, lint, and compatibility checks were rerun after those adjustments, and remote CI checked the final commit.

Maintenance and release decisions

Future changes must continue to establish operation identity, visible failure, truthful per-object outcomes, complete terminal counters, and cancellation that releases its own resources. Neither an API returning Completed nor an object existing at the destination can replace those checks.

The code verdict is GO, and the delivery facts are main integration and passing CI. A formal release still requires selecting a tag, verifying packages and images, and establishing that deployments contain the repair. The scanner-dependent MRF recovery delay, the open cross-pool issue, and the externally reported 405 storm not reproduced on current SILO remain part of this decision record.