Recover after Hardware Failure

• 1: Drive Failure Recovery
• 2: Node Failure Recovery
• 3: Site Failure Recovery

1 - Drive Failure Recovery

MinIO supports hot-swapping failed drives with new healthy drives. MinIO detects and heals those drives without requiring any node or deployment-level restart. MinIO healing occurs only on the replaced drive(s) and in most cases has minimal or negligible impact on deployment performance.

MinIO healing ensures consistency and correctness of all data restored onto the drive.

The following steps provide a more detailed walkthrough of drive replacement. These steps assume a MinIO deployment where each node manages drives using /etc/fstab with per-drive labels as per the documented prerequisites.

1) Unmount the failed drive(s)

Unmount each failed drive using umount. For example, the following command unmounts the drive at /dev/sdb:

umount /dev/sdb

2) Replace the failed drive(s)

Remove the failed drive(s) from the node hardware and replace it with known healthy drive(s). Replacement drives must meet the following requirements:

• XFS formatted and empty.
• Same drive type (e.g. HDD, SSD, NVMe).
• Equal or greater performance.
• Equal or greater capacity.

Using a replacement drive with greater capacity does not increase the total cluster storage. MinIO uses the smallest drive’s capacity as the ceiling for all drives in the Server Pool.

The following command formats a drive as XFS and assigns it a label to match the failed drive.

mkfs.xfs /dev/sdb -L DRIVE1

MinIO strongly recommends using label-based mounting to ensure consistent drive order that persists through system restarts.

3) Review and Update fstab

Review the /etc/fstab file and update as needed such that the entry for the failed drive points to the newly formatted replacement.

• If using label-based drive assignment, ensure that each label points to the correct newly formatted drive.
• If using UUID-based drive assignment, update the UUID for each point based on the newly formatted drive. You can use lsblk to view drive UUIDs.

For example, consider

$ cat /etc/fstab

  # <file system>  <mount point>  <type>  <options>         <dump>  <pass>
  LABEL=DRIVE1     /mnt/drive1    xfs     defaults,noatime  0       2
  LABEL=DRIVE2     /mnt/drive2    xfs     defaults,noatime  0       2
  LABEL=DRIVE3     /mnt/drive3    xfs     defaults,noatime  0       2
  LABEL=DRIVE4     /mnt/drive4    xfs     defaults,noatime  0       2

Given the previous example command, no changes are required to fstab since the replacement drive at /mnt/drive1 uses the same label DRIVE1 as the failed drive.

4) Remount the Replaced Drive(s)

Use mount -a to remount the drives unmounted at the beginning of this procedure:

mount -a

The command should result in remounting of all of the replaced drives.

5) Monitor MinIO for Drive Detection and Healing Status

Use mc admin logs command or journalctl -u minio for systemd-managed installations to monitor the server log output after remounting drives. The output should include messages identifying each formatted and empty drive.

Use mc admin heal to monitor the overall healing status on the deployment. MinIO aggressively heals replaced drive(s) to ensure rapid recovery from the degraded state.

6) Next Steps

Monitor the cluster for any further drive failures. Some drive batches may fail in close proximity to each other. Deployments seeing higher than expected drive failure rates should schedule dedicated maintenance around replacing the known bad batch. Consider using MinIO SUBNET to coordinate with MinIO engineering around guidance for any such operations.

2 - Node Failure Recovery

If a MinIO node suffers complete hardware failure (e.g. loss of all drives, data, etc.), the node begins healing operations once it rejoins the deployment. MinIO healing occurs only on the replaced hardware and does not typically impact deployment performance.

MinIO healing ensures consistency and correctness of all data restored onto the drive.

The replacement node hardware should be substantially similar to the failed node. There are no negative performance implications to using improved hardware.

The replacement drive hardware should be substantially similar to the failed drive. For example, replace a failed SSD with another SSD drive of the same capacity. While you can use drives with larger capacity, MinIO uses the smallest drive’s capacity as the ceiling for all drives in the Server Pool.

The following steps provide a more detailed walkthrough of node replacement. These steps assume a MinIO deployment where each node has a DNS hostname as per the documented prerequisites.

1) Start the Replacement Node

Ensure the new node has received all necessary security, firmware, and OS updates as per industry, regulatory, or organizational standards and requirements.

The new node software configuration must match that of the other nodes in the deployment, including but not limited to the OS and Kernel versions and configurations. Heterogeneous software configurations may result in unexpected or undesired behavior in the deployment.

2) Update Hostname for the New Node

Optional This step is only required if the replacement node has a different IP address from the failed host.

Ensure the hostname associated to the failed node now resolves to the new node.

For example, if https://minio-1.example.net previously resolved to the failed host, it should now resolve to the new host.

3) Download and Prepare the MinIO Server

Follow the deployment procedure to download and run the MinIO server using a matching configuration as all other nodes in the deployment.

• The MinIO server version must match across all nodes
• The MinIO service and environment file configurations must match across all nodes.

4) Rejoin the node to the deployment

Start the MinIO server process on the node and monitor the process output using mc admin logs or by monitoring the MinIO service logs using journalctl -u minio for systemd managed installations.

The server output should indicate that it has detected the other nodes in the deployment and begun healing operations.

Use mc admin heal to monitor overall healing status on the deployment. MinIO aggressively heals the node to ensure rapid recovery from the degraded state.

5) Next Steps

Continue monitoring the deployment until healing completes. Deployments with persistent and repeated node failures should schedule dedicated maintenance to identify the root cause. Consider using MinIO SUBNET to coordinate with MinIO engineering around guidance for any such operations.

3 - Site Failure Recovery

MinIO can make the loss of an entire site, while significant, a relatively minor incident. Site recovery depends on the replication option you use for the site.

Site replication healing automatically adds IAM settings, buckets, bucket configurations, and objects from the existing site(s) to the new site with no further action required.

You cannot configure site replication if any bucket replication rules remain in place on other healthy sites. Bucket replication is mutually exclusive with site replication.

If you are switching from using bucket replication to using site replication, you must first remove all bucket replication rules from the healthy site prior to setting up site replication.

Restore an Unhealthy Peer to Site Replication

Site replication keeps two or more MinIO deployments in sync with IAM policies, buckets, bucket configurations, objects, and object metadata. If a peer site fails, such as due to a major disaster or long power outage, you can use the remaining healthy site(s) to restore the replicable data.

The following procedure can restore data in scenarios where site replication was active prior to the site loss. This procedure assumes a total loss of one or more peer sites versus replication lag or delays due to latency or transient deployment downtime.

1. Remove the failed site from the MinIO site replication configuration using the mc admin replicate rm command with the --force option.

   The following command force-removes an unhealthy peer site from the replication configuration:

   mc admin replicate rm HEALTHY_PEER UNHEALTHY_PEER --force

   • Replace HEALTHY_PEER with the alias of any healthy peer in the replication configuration
   • Replace UNHEALTHY_PEER with the alias of the unhealthy peer site

   All healthy peers in the site replication configuration update to remove the unhealthy peer automatically. You can use the mc admin replicate info command to verify the new site replication configuration.

2. Deploy a new MinIO site following the site replication requirements.

   • Do not upload any data or otherwise configure the deployment beyond the stated requirements.
   • Validate that the new MinIO deployment functions normally and has bidirectional connectivity to the other peer sites.
   • Ensure the new site matches the server version on the existing peer sites
   Caution

   Warning

   The mc admin replicate rm --force command only operates on the online or healthy nodes in the site replication configuration. The removed offline MinIO deployment retains its original replication configuration, such that if the deployment resumes normal operations it would continue replication operations to its configured peer sites.

   If you plan to re-use the hardware for the site replication configuration, you must completely wipe the drives for the deployment before re-initializing MinIO and adding the site back to the replication configuration.

3. Add the replacement peer site to the replication configuration.

   Use the mc admin replicate add command to update the replication configuration with the new site:

   mc admin replicate add HEALTHY_PEER NEW_PEER

   • Replace HEALTHY_PEER with the alias of any healthy peer in the replication configuration
   • Replace NEW_PEER with the alias of the new peer

   All healthy peers in the site replication configuration update for the new peer automatically. You can use the mc admin replicate info command to verify the new site replication configuration.

4. Resynchronize the new peer with mc admin replicate resync.

   mc admin replicate resync start HEALTHY_PEER NEW_PEER

   • Replace HEALTHY_PEER with the alias of any healthy peer in the replication configuration
   • Replace NEW_PEER with the alias of the new peer

5. Validate the replication status.

   Use the following commands to track the replication status:

   • mc admin replicate status - provides overall status and progress of replication
   • mc replicate status - provides bucket-level and global replication status

Active Bucket Replication Resynchronization

For scenarios where bucket replication was in place prior to the failure, you can use mc replicate resync to restore data to a new site. Create a new site to replace the failed deployment, then synchronize the data from an existing, healthy, bucket replication-enabled deployment to the new site.

1. Deploy a new MinIO site.
2. Set up IAM and users as needed.
3. On the site with data, create a new remote target using the mc admin bucket remote add command and record the ARN from the output.
4. From the site with the data, use the mc replicate resync start command with the ARN from the previous command to rebuild the bucket on the new site.
5. Wait for re-synchronization to complete (use mc replicate resync status to check).
6. Set up bucket replication rule(s) from the new MinIO site to the existing target bucket(s).
7. (Optional) Delete the bucket replication rules from the target deployment(s) to restore an active-passive replication scenario.

Passive Bucket Replication Resynchronization

Bucket replication can directly restore the site contents by performing a replication from the target bucket(s) to a new MinIO site.

As a passive process, bucket replication may not perform as quickly as desired for a site recovery scenario.

Bucket replication relies on the standard replication scanner queue, which does not take priority over other processes. For recovery procedures with stricter SLA/SLO, use the active bucket replication process with mc replicate resync command as described above.

Bucket replication rules copy the object, its version ID, versions, and other metadata to the target bucket. MinIO can restore the object with all of these attributes to a new MinIO site if bucket replication had already been in use prior to the site loss.

1. Deploy a new MinIO site.

2. Set up IAM and users as needed.

3. On the remaining target bucket deployment(s), create bucket replication rule(s) for each bucket to the new MinIO site.

4. Wait for replication to complete.

5. Set up bucket replication rule(s) from the new MinIO site to the existing target bucket(s).

6. (Optional) Delete the bucket replication rules from the target deployment(s) to restore an active-passive replication scenario.

   Do not delete the bucket replication rules from the deployments used to recover data if you prefer to keep an active-active replication between the buckets. In active-active replication, changes to the objects at either location affect the objects at the other location.

Mirroring

MinIO’s mirroring copies an object from any S3 compatible storage system.

Mirroring only copies the latest version of each object and does not include versioning metadata, regardless of the source. You cannot restore those attributes with this method.

Use mc mirror in situations where you need to restore only the latest version of an object. Use bucket replication or site replication where those methods were already in use if you are copying from another MinIO deployment and wish to restore the object’s version history and version metadata.

1. Deploy a new MinIO site.
2. Set up IAM and users as needed.
3. Create buckets on the new site.
4. Use the mc cp CLI command to copy the contents from the mirror location to the new MinIO site.