Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence

#77 is a reproduced site-replication correctness defect. A receiver replaces source time with arrival time and may then reject a genuinely newer deletion. Some configuration types stop exporting their timestamp after deletion, preventing heal from recovering a delete missed during an outage. Adding a DELETE branch alone cannot solve both problems.

Merge status (2026-09-12): the repair and research archive were merged into main through PR #180, commit 48ec10312. #77 is closed. All nine pre-merge CI checks passed.
Review boundary: the plan went through four Claude Code Opus 5 Max review rounds, passing the final two. Full implementation review, remediation review, and focused final acceptance returned GO_WITH_NONBLOCKING_NOTES in all three rounds. Final blockers are zero; the requested full cmd and final lint checks have now passed.
Applicability: this page describes the repair now included in main. Check the specific version of downloads and running installations; full deletion recovery still requires every node to be upgraded and deletion export to be enabled consistently.

Existing work and scope

The earlier release notes, security hardening record, and Server compatibility page already document the #77 deletion limitation. They do not provide a complete record of its state model, alternatives, or verification boundaries. This page supplies that reasoning.

The source repository archive retains the original reproduction, plan versions, final reports and invocation identities for all seven review rounds, finding dispositions, executed test logs, source/binary hashes, and a rerunnable two-site driver. Raw model reasoning streams, binaries, and temporary lab volumes are excluded. Original artifacts and copies with normalized workstation paths and document links have separate hashes.

This repair covers Policy, Tags, SSE, Quota, Versioning, and Object Lock. Lifecycle/expiry has its own merged-payload time semantics; CORS keeps its separate mechanism. Notification, IAM, object replication, MRF, resync, and public counters are not rewritten. Object replication reliability has a separate record.

The supported release target is the maintained silo, silo-console, mc, and silo-pkg stack. Compatibility with unmodified upstream MinIO/MC remains best effort and does not require downgrading maintained components or recreating dependency forks.

What the reproduction established

The original regression reproduced on both ErasureSD and 16-disk Erasure ObjectLayers. A peer PUT originated at a past time T, but the receiving disk stored current arrival time. A subsequent DELETE carrying T+1 minute was rejected as older. RPC success alone therefore cannot establish correct final state.

Three further entry-point defects matter: an old bulk event can overwrite a newer field; a request that checks state before queuing for the lock can act on an obsolete decision; and remote Tag heal omits UpdatedAt. Each requires a repair at the actual entry point, beyond changing source selection in heal.

The facts a field needs

Reuse the existing payload, field UpdatedAt, and bucket Created. No disk format, SDK field, or persisted deployment ID is added.

Empty Versioning and Object Lock events remain no-ops. Sharing a helper must not give these types deletion semantics. Zero field times use Created as a comparison baseline; this does not turn historical emptiness into a new delete.

Ordering first gives real states priority over baselines, then compares source time between real states. At the same time, a deletion wins over a live value. Conflicting live values at the same rank use a stable content key, with the lexicographically greater key winning. An identical effective state causes neither a save nor a notification. Live baselines also use content-key ordering; a later creation default cannot outrank a real change.

This is deterministic conflict resolution. It does not mean a lexicographically greater configuration better expresses business intent. Operators must still choose and resubmit the intended value after conflicting concurrent changes.

Comparison must match persistence

Policy sets are map-backed, so ordinary JSON encoding can depend on iteration order. The Server sorts the complete JSON tree of an already validated policy, including Statement, Action/NotAction, Resource/NotResource, Principal, and Condition. Sid and numeric precision are retained. This adds no syntax unsupported by the existing parser.

That parser accepts NotAction and NotResource, but the original structure’s required Action/Resource encoding can fail on the corresponding empty sets. Explicit field encoding addresses this, and Policy GET/admin export use it too. The purpose is to keep an accepted policy readable, beyond making its comparison key deterministic.

Quota keys use the existing parsed representation encoded as JSON. {}, JSON null, and valid zero-quota documents remain live documents, not implicit deletion events. An empty Policy instead follows the established peer deletion interpretation: a valid empty-policy PUT succeeds, and a subsequent GET returns the existing NotFound response.

XML configurations use the bytes of a valid document. There is no general XML canonicalization layer. Versioning needs one exception: apply the existing Object Lock constraint before comparing the effective document that will actually be persisted. Otherwise comparison can accept a value that Save rewrites, causing heal to send it again next time.

Lock the decision as well as the save

The six fields share one .metadata.bin record. Loading raw state, validation, comparison, modification, and saving must all happen under the existing metadata.lock. Comparing outside the lock still permits stale decisions; separate field locks would allow whole-record read/modify/write operations to overwrite one another.

A local write allocates its time inside the lock:

max(current UTC time, Created + 1ns, current field time + 1ns)

This lets a local correction advance beyond an already stored future field time. Timestamped peer events retain their original time. Identical or older states return without a write.

Bulk processing handles only explicitly supplied fields, validates them, and saves at most once. Omission preserves a field; explicit null follows its type’s semantics. An invalid field cannot leave half the bulk update persisted. Import assigns a common time for the selected six-type fields under the final per-bucket commit lock. Disk state and outbound events use that commit’s final snapshot, not a later read combined with an earlier time. A normalized empty Policy uses the existing dedicated deletion event so bulk omitempty cannot lose it.

The save helper returns its normalized snapshot. Public Update/Delete signatures stay unchanged, while other metadata retains its existing processing and notification behavior.

Adoption must not manufacture a deletion

Adoption can change Created. Moving it earlier while retaining an empty field’s old default timestamp makes an empty baseline appear to be a real deletion. Moving it later can invalidate historical initial values.

The narrow repair, under the existing adoption lock, rebases only these six fields whose previous time was zero or equal to old Created. Actual update and deletion times remain unchanged. An empty payload alone is not proof of a default, and existing-bucket configuration protection is not redesigned. Genuinely different bucket generations still require operator intervention.

One rule across the real entry points

Initial synchronization retains its existing five-type send path. Versioning is still initialized through MakeBucketHook and aligned through heal. No extra initialization path is added merely to make the table symmetric.

Heal filters valid candidates before selecting the maximum, instead of seeding from the first map entry and only then filtering defaults. Public mismatch counters cannot be the sole gate: equal content with different source times still needs synchronization. Conversely, equal effective states need no further write or RPC.

Why rollout needs a default-off switch

The startup setting MINIO_SITE_REPLICATION_METADATA_TOMBSTONES defaults to off. It controls visibility of newly exposed deletion information and does not detect remote capability.

Old implementations cannot safely consume all newly exposed deletion information. For example, old Quota heal can clear the payload while retaining an already parsed cache value. An instruction to upgrade does not itself isolate this rolling-upgrade window, so the default stays off.

Upgrade every node at every participating site to a build containing the repair, ensure consistent settings within each site, and drain old requests. Then set on consistently and restart. Before a downgrade, first set off and restart every fixed node, then roll back the software. The old software’s original defects return with it.

While off, hidden Tags/SSE/Quota tombstones can cause repeated stale heal RPCs that a fixed receiver rejects. A subsequent heal with zero RPCs is expected only when complete state is visible and stable.

Retained and rejected alternatives

The initial implementation adds 729 and removes 692 production Go lines, a net increase of 37, chiefly replacing duplicated apply and heal branches. Line count does not establish minimality. Necessity must connect each mechanism to a concrete failure; sufficiency must cover every real entry point; minimality asks which failure returns if a mechanism is removed.

Validation and its limits

The environment is local go1.27.1 darwin/arm64. Four groups of pre-implementation audit tests failed on the unfixed baseline. The resulting regression suite passes on both ObjectLayers; its main entry points are in cmd/site-replication-metadata{,-heal,-gate}_test.go. The baseline audit log retains the original failures alongside the passing existing tests.

Local evidence contains baseline failures, test logs, a rerunnable two-site driver, metadata snapshots, and source/binary SHA-256 identities. The first two-site binary reports c8f264f79 + dirty; its actual Go build-info and SHA-256 have now been recorded. After review repairs and cleanup, runs were repeated on binaries built from clean 62cf066ff and fcbb93e89. Actual --version, Go build metadata, and SHA-256 identities are retained with the baseline binary built from 5c5765816. Final 461e9a721 differs from fcbb93e89 only in test formatting and equivalent conditional syntax; that diff is recorded separately.

The table above records local tests and isolated-process observations. Remote integration evidence is separate: all nine checks on PR #180 passed, comprising six Go CI jobs, DCO, vulnerability analysis, and release-pipeline validation. The merged main tree was verified to match the PR merge tree that passed these checks. This is not proof of real Linux multi-node cluster behavior, official release artifacts, or production deployment.

Adversarial review record

The plan used actual Claude Code claude-opus-5 --effort max for four rounds. The first two drove corrections to state comparison, committed snapshots, historical baselines, and import boundaries. The last two returned GO_WITH_NONBLOCKING_NOTES with zero pre-implementation blockers. Plan approval is not proof that implementation is correct.

The separate implementation review pinned 4089113e3 and used the same model and effort to inspect the complete diff, production call chains, formal tests, and runtime evidence independently, challenging sufficiency, minimality, rollout safety, and evidence identity. Its verdict was GO_WITH_NONBLOCKING_NOTES: no unconditional blocker, one conditional blocker, and nine further findings. It confirmed the core convergence mechanism and found no counterexample to ordering, deletion, or duplicate suppression within the declared contract. Three findings were real defects the change had introduced and were repaired in 62cf066ff. The subsequent fcbb93e89 closes the diagnostic gap when no valid source exists and adds an initial-sync regression.

A stub is worth calling out separately. The original recovery test injected an object layer whose creation probe returned the expected time, so it passed against code that could never behave that way in production. The replacement stamps the bucket directory on every local drive and drives the real object layer, and it fails on the unrepaired code.

The second review pinned 62cf066ff, again using actual claude-opus-5 --effort max. It returned GO_WITH_NONBLOCKING_NOTES with zero conditional or unconditional blockers. It retraced production paths, checked the author’s F1/F2 reproduction results from formal tests with old production code overlaid, and revised the first review’s assessment of the Policy encoder. The reviewer did not execute the tests.

Diagnostic regressions now use the existing logger target to check warning level, distinct reasons, repeated calls, and zero RPCs. The initial-sync test drives a real source ObjectLayer and the complete outbound sequence; its peer only acknowledges RPCs. That proves outbound content. The real two-process experiment supplies a separate level of evidence, and the two must not be conflated.

The third focused acceptance pinned final production code fcbb93e89 and again returned GO_WITH_NONBLOCKING_NOTES, with zero blockers. It also inspected the test-only style diff in 461e9a721 and confirmed equivalent semantics. Full cmd and lint were still running when the reviewer read their logs; both subsequently exited with code 0. Test formatting caused lint to fail at fcbb93e89 itself; corrected 461e9a721 is the delivery baseline that satisfies the required checks.

Three nonblocking improvements remain outside this repair: heal diagnostics can show a zero source time after decode/parse failure; the initial-sync unit test does not execute the local peer branch and therefore does not prove that recovered Created is persisted locally (that production path was reviewed); and strict system-log capture could need isolation if concurrent background logging is introduced. No additional production accessor or test hook was added for these points. A counterfactual failure proves the assertion actually reached, such as empty-baseline noise. Warning-level and per-reason deduplication assertions pass after repair; that is not a claim that each was separately demonstrated failing on old code.

Operational boundaries that remain

1. Historical timestamp pollution is not reconstructable. Arrival-time replacement and old untimestamped events have lost source facts. Installing the repair cannot recover their true historical order. Inspect every site and resubmit the intended configuration or deletion at the authoritative site.
2. Zero-time typed events remain compatible. They use monotonic local time and emit legacy-zero; the existing zero-time constraint for bulk remains. These events are outside the timestamped-source convergence guarantee.
3. Bucket identity conflicts are not merged automatically. Resolve generation differences first. Events before target Created are not applied; unknown creation time is recovered only from a real physical bucket. Unknown or missing buckets are not written. The recovered value is a physical approximation from bucket-directory modification time. It can change with top-level entries, differ across drives, and be later than actual creation. Older source events are still skipped; one event is insufficient to lower the bucket identity. A successful configuration write or initial site sync records the recovered value. Until then, status reports the unknown time and periodic healing skips the bucket in both directions.
4. A physical clock is not a causal clock. Local correction can advance beyond an already known future field time, but cannot infer the business intent of all concurrent writes.
5. Diagnostics are bounded; success does not mean applied. legacy-zero, before-created, indeterminate, unreachable, and peer-error reuse LogOnceIf with stable keys and error text, details in attributes, and existing hourly cleanup. Each reason keeps its own key, so one condition cannot deduplicate another away. Empty baselines and peers that do not yet have the bucket stay quiet. Invalid existing states emit indeterminate even when no valid source can be selected, without causing an RPC. Normal duplicates and older events remain quiet. This is a per-bucket/field/reason bound: both unreachable-peer warnings and indeterminate-state warnings can grow with bucket and populated-field count, not a fixed site-wide limit.
6. Code, integration, and release require separate acceptance. Issue state, Server version, images, packages, published documentation, and production settings each need their own evidence. The main-branch merge recorded here does not establish that release artifacts or production installations have been upgraded.