ZFS: Replace a failed disk

Prologue

This guide works for Proxmox VE 5.4 and higher.

In this guide we will replace a failed boot disk, and reinstall the bootloader on it.

Warning: Running the wrong command here can definitely destroy your system. Make sure you have recent and external backups.

This guide uses [source] and [target] as placeholders for devices in /dev/disk/by-id/, as seen in the screenshots.

If you need to replace a disk that's not used for booting, you can skip the proxmox-boot-tool commands.
Also adjust the partitions in the commands; you can use lsblk -f /dev/disk/by-id/[target] on your target disk, to get an overview.

Determine the bootloader

Run proxmox-boot-tool status in your Proxmox VE console:

In this image we can see, that we are using grub.

If your system was installed with secure boot enabled, it'll use grub.

And if it was installed without secure boot enabled and on EFI, it'll use systemd-boot.

This is important for later, so keep your bootloader in mind.

zpool status

Let's check the zpool status, running zpool status:

We can clearly see that our scsi0 disk failed.

Set failed disk to offline

Let's set this drive to offline, this will make sure it won't come back while we are replacing it.

Command: zpool offline rpool /dev/disk/by-id/[source]. (The failed drive)

We can use zpool status to see the change:

Shutdown and replace disk

Turn off your VE host, and replace the failed disk with a new one.

It should be the same size or bigger.

Once done, start up the host again and connect to the console.

Observe your disks by running ls -l /dev/disk/by-id/.

In this guide, the drive replacing the unavailable one has the same by-id name. It will likely be different on real hardware.

Copy partition table

To copy the partition table to your new disk run following command:

Command: sgdisk /dev/disk/by-id/[source] -R /dev/disk/by-id/[target].

Then run sgdisk --randomize-guids /dev/disk/by-id/[target] to randomize the GUIDs:

Info: If you use a different ZFS-RAID than a mirror, you can just pick any working boot disk as [source].

zpool replace

Now we want to replace the disk of the partition 3 in our ZFS pool.

Command: zpool replace rpool /dev/disk/by-id/[source]-part3 /dev/disk/by-id/[target]-part3.

We can observe the resilvering process by running zpool status.

Updating the bootloader

First we need to format partition 2, which is the EFI system partition.

Command: proxmox-boot-tool format /dev/disk/by-id/[target]-part2.

Then we issue following command. If your system uses systemd-boot, simply run the following command without the grub parameter, and it will install systemd-boot instead.

Command: proxmox-boot-tool init /dev/disk/by-id/[target]-part2 grub.

🎉 Congratulations, if everything worked out, you should be able to boot from both drives in our configuration now.