Overwrite Pool Affinity: Why #245 Was Not Adopted

Decision, October 24, 2026: do not merge the tested pool-affinity changes (PR #245). The experiment addressed a real space-efficiency problem, but neither the broad policy nor the narrower unversioned-only policy achieved an acceptable performance trade-off. Both experimental branches (codex/healthy-pool-affinity and codex/unversioned-pool-affinity) were retained for reference while keeping main unchanged.

Denis Babchuk proposed keeping overwrites in the pool that already holds the object. Review identified and fixed a request-specific write-quorum bug, tightened the health policy, and validated the behavior. Performance measurements then changed the recommendation: the broad policy made deep version histories substantially more expensive, and a second iteration restricted to ordinary unversioned overwrites still regressed small-object writes.

The subsequent multi-pool deletion fix (#257) was merged independently on October 24: while retaining capacity-weighted allocation, it cleans up non-authoritative copies before deleting the authoritative copy so a mid-delete failure cannot expose an overwritten value.

The problem: an overwrite can leave its old copy in another pool

Consider an unversioned key, reports/latest.json, in pool A. A subsequent PUT writes its replacement to pool B. Reads select the newest logical object, while the old physical copy in A continues to occupy space because ordinary overwrites do not clean up obsolete copies across other pools.

The baseline looks up the existing object and then sends a readable result through capacity-weighted pool selection again. Depending on pool usage, repeated writes can eventually leave a pool-local record (xl.meta) in both pools.

This behavior has a specific history:

• Upstream MinIO issue #21087 reported that single-pool uploads could bypass the minimum free-space threshold. PR #21115, merged on July 19, 2025 as 83b2ad418b, enforced capacity checks by reselecting a pool for readable existing objects, which dropped their earlier placement affinity.
• SILO issue #199 and PR #207 evaluate conditional writes against the current logical object across pools while keeping the capacity-weighted allocation policy.
• Pool migration uses separate safeguards against stale payloads and delete-marker loss; see Multi-Pool Object Consistency.

The space benefit of overwrite affinity applies specifically to multi-pool deployments with repeated overwrites of the same unversioned keys. Single-pool deployments have no cross-pool duplicates, new keys still require capacity-weighted allocation across pools, and versioned buckets must retain their historical versions regardless of which pool stores them.

What the original PR changed

The proposal reused the existing object lookup and lock boundary. If the latest readable object was already present, its pool became the preferred destination, subject to write quorum and the existing hard free-space check. New keys retained capacity-weighted allocation. When the existing object’s set could not accept a write, allocation excluded that pool before considering the remaining pools.

The production change in cmd/erasure-server-pool.go added 73 lines and removed 3 lines, accompanied by 331 lines of placement tests, covering PUT and multipart initialization on the shared allocation path.

Two capacity rules apply during selection:

• Soft allocation reserve (15%): when alternative pools exist, the allocator filters out pools above roughly 85% usage so emptier pools fill first.
• Hard space check (hasSpaceFor): checks write size, per-drive capacity, and other constraints against diskFillFraction (0.99).

Affinity bypasses the soft reserve when the existing object’s pool is healthy and has room, while preserving the hard capacity check. After a pool expansion, existing keys continue updating in their older, fuller pool rather than migrating to the new pool on overwrite.

Simply deleting the condition that reselects existing objects would lose the capacity checks and fallback introduced in 83b2ad418b. Checking space alone is also insufficient: an erasure set can still have read quorum for an old object while lacking enough online drives to reach write quorum for its replacement. Because pool selection runs before consuming the request body, an unreplayable upload stream cannot be transparently retried in another pool after a write failure.

Correctness review and implementation improvements

Write quorum must match the request

The original helper inferred quorum from the STANDARD storage class for every write. A controlled two-pool test exposed a regression when overwriting an object with REDUCED_REDUNDANCY (RRS):

The helper required 12 writers for EC:4 and selected the original pool, whereas the actual EC:2 write required 14 online drives and failed with InsufficientWriteQuorum (HTTP 503). Because lower parity leaves more data shards, RRS requires more online drives for a write. The baseline’s reserve filter skipped the 90%-full pool, selected the healthy second pool, and succeeded across all four pool/entry-point combinations (12/12 attempts failed on the unpatched PR).

The inverse mismatch occurred with PUT’s internal MaxParity option: a write eligible for EC:8 was evaluated against EC:4 and unnecessarily moved to another pool.

A healthy preference, with explicit fallback

The updated implementation passed the request’s storage class and PUT MaxParity into placement, matching the writer’s availability/capacity parity rules (including the extra drive required when data and parity shard counts are equal) while preserving multipart initialization parity semantics:

Health and capacity checks shared a single set-level DiskInfo snapshot. A drive counted as a usable writer only if it was online and returned a positive capacity result; drives with a failed DiskInfo call were treated as unhealthy.

Regression tests were added for request parity, known disk faults, degraded sets, scan counts, heterogeneous set sizes, latest-object selection, and existing multipart sessions. The multipart fallback test was also updated to seed its own initial object rather than reusing a prior PUT target that had already moved pools.

The second design: only ordinary unversioned overwrites

After the first performance round showed high version-history rewrite costs, a second branch (codex/unversioned-pool-affinity) narrowed the policy to ordinary unversioned overwrites:

Eligibility checked both the bucket’s versioning configuration and the request options. A version-enabled bucket with prefix-excluded unversioned writes, or a version-suspended bucket with mixed history, remained on the baseline allocator.

Functional validation, separated by revision

The targeted regression filter was:

^(TestPoolsOverwrite|TestPoolsMultipartOverwrite|TestSinglePoolOverwrite|TestPoolsConditionalPut|TestSinglePoolConditionalPut|TestPoolsMultipartCondition)

Benchmark method and what the numbers mean

Both performance rounds ran on m3 (AMD Ryzen AI MAX+ 395, 32 logical CPUs, 61 GiB RAM, Linux 7.0.0-38, CT2000P310SSD8 NVMe with XFS). Two pools each used 16 directories and one erasure set (EC 12+4), all backed by the same XFS filesystem on the shared NVMe SSD.

Both variants were built with Go 1.27.3 (linux/amd64, CGO_ENABLED=0). The server ran on CPUs 4–11 (GOMAXPROCS=8), and the minio-go v7.4.1 benchmark client ran on CPUs 12–15 (GOMAXPROCS=4) over loopback HTTP.

• Throughput: completed object operations per second (for multipart, one operation includes initialization, two part uploads, and completion).
• P50 / P99: medians across runs of each run’s 50th and 99th percentile request latencies.
• CPU cost: server CPU time per completed operation.
• Block I/O: process-accounted reads and writes from /proc/PID/io.
• Active-bucket space: object files, metadata, and directory overhead excluding temporary files.

Round one: the full policy cost more than expected

Round one compared the full candidate (0aa870115d7d) against the baseline (789a2327ba8c) across 54 measured runs (243,718 completed operations, 0 failures). Basic workloads ran 3 pairs of 15-second runs, with 2 additional 60-second pairs for new-key and 4 KiB overwrite workloads, 3 fixed-count runs of 4,000 versions on a single seeded object, and a 4-run ABBA binary-switch sequence on existing dual-copy data:

For ordinary 4 KiB overwrites, both 60-second pairs were slower (−30.7% and −21.8%), as were all three 15-second pairs, even though CPU per operation fell from 5.85 to 5.33 ms and block writes stayed near 128 KiB per operation.

The −30.8% version-history regression had a direct structural cause. A pool-local xl.meta stores the object’s version metadata and inline payload for small objects, and adding a version reserializes and rewrites the entire file. Capacity allocation split the 4,000 versions roughly evenly across both pools (~2,000 versions per xl.meta), whereas affinity concentrated all 4,001 versions into a single pool, doubling the xl.meta rewrite size on every PUT:

• CPU per write: 26.61 ms → 39.29 ms
• Block reads per write: 12.20 MiB → 24.42 MiB
• Block writes per write: 12.27 MiB → 24.49 MiB (cumulative block writes over 4,000 operations rose from 47.9 GiB to 95.7 GiB for ~15.6 MiB of uploaded payload)
• Active data at 4,001 versions: ~48.90 MiB in both variants
• Mean latency over the final 1,000 writes: 18.4 ms → 28.5 ms

Because versioned buckets must keep all historical versions, concentrating them in one pool doubled the metadata rewrite amplification without saving disk space. This motivated narrowing the policy to unversioned overwrites.

Round two: narrowing fixed one cost, but not the acceptance problem

Before running round two (codex/unversioned-pool-affinity vs. 789a2327ba8c), the acceptance threshold for the experiment was set at ≤ 5% ordinary-overwrite throughput loss and ≤ 10% P99 latency growth, with no material P50 or excluded-scope regression.

Four workloads ran 5 pairs each in alternating AB/BA order (5-second warmup and 60-second measurement for timed workloads; 4,000 operations for versioned control; fresh server and data directories per run, no profilers attached):

On the primary 4 KiB overwrite workload, pairwise throughput changes across the five pairs were +11.9%, −14.9%, −1.4%, −10.3%, and −12.8% (median paired change −10.3%; ratio of variant medians −5.7%; P50 +14.4%). Baseline throughput ranged from 141.4 to 202.6 ops/s (CV 13.0%) and narrowed throughput from 146.3 to 176.6 ops/s.

Excluding versioned buckets eliminated the round-one xl.meta write amplification: block writes per version were 12.272 → 12.268 MiB, block reads 12.201 → 12.197 MiB, and CPU 26.98 → 27.26 ms (in the versioned control, the second run of each pair was faster regardless of binary order, and new-key control pairs varied from −44.3% to +15.3% due to shared-SSD order effects).

Restore the same starting data before each run

Because fresh runs let the two placement policies build different on-disk layouts over time, an additional 8-run fixture test restored an identical snapshot before each run—either one pool-local copy per key (256 records) or copies in both pools (512 records)—followed by a 30-second settle period, 5-second warmup, and 60-second measurement:

The single-copy pairs changed by −0.9% and −23.5%. The dual-copy pairs declined by −22.6% and −26.5%, with P50 rising +111.9% and P99 rising +36.9%, while leaving all 512 existing records on disk. On clusters that already have cross-pool duplicates, the narrowed policy paid a steep write penalty without reclaiming any existing space.

Diagnosis and the I/O optimization attempt

Phase timing measured the new health check at 0.014–0.016 ms per call, compared with milliseconds for existing-object lookup and tens of milliseconds for erasure writes. In round two, 4 KiB overwrite CPU time fell from 5.75 to 5.36 ms per operation while throughput dropped, pointing to I/O and scheduling wait rather than CPU overhead.

Go execution traces showed syscall waits concentrated in RenameData → writeAllInternal → write, alongside read and goroutine scheduling waits.

A follow-up experiment routed small metadata writes (~791 bytes, below the 4 KiB alignment threshold where CopyAligned disables O_DIRECT) from the Linux O_DSYNC path to writeAllDirect, turning the write into a buffered write followed by fdatasync. Across five runs, narrowed affinity plus the I/O change averaged 179.4 ops/s versus the baseline’s 167.9 ops/s (+6.9%, P99 −1.5%, pairwise range −19.2% to +35.9%). However, when the same I/O change was applied to both baseline and candidate in two exploratory pairs, baseline-plus-I/O reached 212.5 ops/s versus 185.2 ops/s for affinity-plus-I/O (−12.8%, pairwise +37.7% and −49.1%). Because changing small-metadata sync paths affects the entire storage engine and did not isolate the affinity regression, the overlay was discarded.

What it buys, and what it costs

In fresh two-pool tests, 256 ordinary 4 KiB keys ended with 512 → 256 pool-local records and active logical size of 6.18 → 3.09 MiB. The 1 MiB workload ended with 128 → 64 records and 171.56 → 85.78 MiB.

At the cluster level, if a fraction $f$ of logical data would otherwise accumulate two copies across two pools, avoiding the second copy reduces steady-state space by $f / (1 + f)$ (for example, ~9.1% at $f = 10%$). Per-request write volume is unchanged: ordinary 4 KiB overwrites generated ~128 KiB of block writes per operation in both variants.

Final decision and conditions for revisiting it

1. Full affinity (codex/healthy-pool-affinity): rejected because concentrating version histories in one pool doubled xl.meta rewrite I/O and cut 4,000-version throughput by 30.8% without saving space.
2. Narrowed unversioned affinity (codex/unversioned-pool-affinity): rejected because ordinary 4 KiB overwrites still regressed by 5.7% on fresh data (median paired change −10.3%) and by 24.5% (P50 +111.9%) on restored dual-copy fixtures.

We therefore kept the mainline allocator unchanged and archived the benchmark records (54 runs / 243,718 operations in round one; 53 runs / 495,745 operations in round two). Any future revisit should start from production duplicate-space measurements and multi-disk hardware benchmarks, and evaluate whether background stale-copy cleanup or an opt-in placement policy better addresses the workload.

Source and evidence identification

Public references: PR #245, upstream PR #21115, Multi-Pool Object Consistency, and the baseline storage implementation in cmd/erasure-server-pool.go, cmd/xl-storage-format-v2.go, and cmd/xl-storage.go.