Terminology

In this guide we will use following terminology:

VM

Virtual machine - The virtual computer running on the Proxmox VE Host.

Node

A node describes a Proxmox server within a cluster.

Cluster

Multiple Proxmox VE instances that are joined using the cluster functionality.

Replication (ZFS)

Replication permanently copies a data set to another node so that failover can be carried out and it also speeds up migrations.

SAN

Storage Area Network - A storage network that can be accessed directly, for example using iSCSI.

HA

High availability. An automated failover that will keep your virtual machines or containers alive, even when a node dies.

In combination with a SAN or replication, a node can die and the machine will start up on another node.

Failover

An automatic fail-safe mechanism.

Preparation and information

Before we can get started, you need to have following:

• If you only need replication, 2 Proxmox servers will suffice, 3 or more for a HA cluster.
• Make sure all your Proxmox servers are available in the same network.

About the node amount

There's something called a split-brain scenario. If you configure HA with only 2 nodes, you may run into it.

The split-brain scenario occours when both servers can't see each other, so HA will try to start the machines, being unable to communicate the status.

That could result in multiple equal IP addresses in the network, as well as inconsistent data and broken databases.

If you use 3 nodes or more, your cluster can avoid this scenario, because two nodes need to confirm the loss of a node, before starting virtual machines.

Can I disconnect a node from a cluster?

The answer is yes, but it's not recommended. If you want a clean setup, reinstall your VE instances.

You can find guides how to accomplish this in the Proxmox documentation.

Network

For best results, make sure all your Proxmox hosts use the same switch and modern network cables.

In most cases a CAT 7 cable will do the job.

Creating the cluster

Click Datacenter in the top left, and select Cluster.

Now you can create a new cluster by clicking Create Cluster.

Enter a useful name that describes your cluster:

When it worked, you will see TASK OK, you can close the task viewer:

Great! Now you have a cluster. With the next steps you will join your nodes to your cluster.

Joining a node

On your primary node where you created the cluster, click "Join Information".

Click Copy Information.

You can now connect to your other Proxmox Servers. Then navigate to the Cluster menu from the Datacenter tab.

Hit "Join Cluster" and paste your information.

Enter the root password of the initial node, and select a link.

Then hit Join '<cluster name>'.

If everything worked out, you should see following:

Reload the page, you may see a certificate warning.

Observe on your first node that you can now manage your secondary node within the cluster.

Do this for all your nodes that should belong to this cluster.

Great, your cluster should look something like this now:

Keep in mind, you still need to configure HA, so your machines will actually do a failover. (3 Nodes or more)

If you followed this guide to do a replication, you're basically almost set.

When you don't have a SAN, you'll want to configure Replication for each machine/container.

Configuring HA (3+ nodes only)

As explained earlier in this guide, only configure following if you have 3 or more nodes in your cluster.

Otherwise you may run into a split-brain scenario which could lead to inconsistent data.

Be aware, that HA works when you have replication configured properly or by using a shared storage, a SAN.

Adding resources

Click on Datacenter on the top left, and navigate to HA.

You will see that there are no resources assigned for HA.

For every virtual machine you should configure HA.

Hit Add and select a VM you want to configure HA for.

You can configure more options like Max. Restart or Max. Relocate.

For most setups the standard values will suffice.

Max. Restart: The maximum number of attempts to restart on a node after it has failed.

Max. Relocate: The maximum number of service relocate tries when a service failed to start.

Groups

With groups you can create zones that only include selected nodes.

You can assign a group when configuring HA for a VM.

This will also enable you to distribute across nodes using priorities.

Keep in mind, if the priority is higher, then that node will be used. If a node isn't selected, then it will be avoided.

restricted: This option makes the assigned resources run on any cluster node if all group members are offline. They will migrate back when a group member comes online.

nofailback: With this you can prevent an automatic migration back when a node comes back healthy.

Here's an example of a HA group which gives highest priority to node 3.

Make sure that you have at least replication set-up, or you have a shared SAN. Otherwise a failover will not work and may require manual fixing later. (See Errors section of this page.)

Replication

Warning: This only works with ZFS based storage.

Replication is great when you don't have a shared storage (SAN).

A Replication will try to keep the data synced across your configured nodes.

For a 2 node cluster this should be configured to allow a fast migration.

Configuring Replication

Click on a VM, then select Replication:

Hit Add, then select your targets, you can repeat this for each node.

Click Create.

Your virtual machine should now replicate.

Migrating

You can either automatically migrate with HA groups, or you can also manually right click a VM and click on Migrate.

This process will be faster if you have a replication on the target node.

Errors

There's multiple things that can go wrong, here's a few things listed:

HA failover without a disk on the target node

This can happen if there was not enough time for the first replication to finish, or simply if it was never set-up.

To resolve this, check if the disk exists on the node where the failed VM is, if it's not there, search it on the other nodes.

Once you know where the disk is, you can go back to the node where the failed VM is, if it's missing on the nodes storage, remove the VM from HA, then delete the disk of the virtual machine associated with it, which doesn't exist on the node.

Now migrate the VM back to the original node, where the disk is. You can see if it's the correct one because the disk is named with the VM id.

Go to the nodes shell and type "qm rescan". You should now be able to assign the disk back to the VM.

Don't forget to re-add the VM to a HA group.

Migration freezes VM

If you use the host type CPU for your VM, and your hardware is different across nodes, then your online migrations could fail.

This is because of the CPU differences.

To prevent this, make sure to use a virtual CPU setting like x86-64-v3 or x86-64-v4, depending on your hardware.

Here's an external article that describes how to determine the correct CPU type: https://www.yinfor.com/2023/06/how-i-choose-vm-cpu-type-in-proxmox-ve.html.