I recently got some questions on a new feature that was introduced in vSAN 7.0u3: “Nested Fault Domains for 2-node vSAN clusters”. With this feature you’re able to provide additional availability on your 2-node clusters: a RAID-1 configuration (mirroring) across the available nodes combined with a RAID-1 configuration on each of the two nodes. In this way you can survive a node failure and failure of a disk group on the remaining node. Read more about the latest updates in 7.0u3 in Duncan Epping’s blog, nested fault domains are also explained here.
In this article we will look at setting up a 2-node vSAN cluster and configure the Nested Fault Domain policy. We will see how vSAN components are placed, and I will als show you what happens after a host failure, with a disk group failure on the remaining host on top of this.
To get Nested Fault Domains for 2-node vSAN cluster up and running you need:
- A 2-node vSAN cluster;
- Each node must have at least 3 disk groups;
- Setup a vSAN policy that leverages the Nested Fault domains capability.
Setup the 2-node vSAN cluster
As you probably know, a 2 node vSAN cluster is a “mini” vSAN stretched cluster setup that includes 2 vSAN nodes and a witness node. Each vSAN and witness node is part of its own failure domain. I’ve prepared two (virtual) ESXi hosts that include the required disk groups and also deployed a witness appliance. Setting up a 2-node vSAN cluster is very simple process, just start the Configure vSAN wizard and select the Two node vSAN Cluster option.
We need three disk groups to make the Nested Fault Domains work, so that means 3 cache devices and (at least) 3 capacity devices:
Now select the witness node that get the 2-node cluster up and running:
After completing the wizard your 2-node vSAN cluster will automatically be configured.
Configure the Nested Fault Domain policy
Now it’s time to setup our Nested Fault Domains policy. Actually this is pretty simple: create a new VM storage policy and select Host mirroring – 2 cluster as site disaster tolerance configuration and select 1 failure – RAID-1 for Failures to tolerate:
You’re now ready to attach this policy to your apps (VMs).
I’ve created a VM called vsan-vm01 with a disk of 16 GB. The Nested Fault Domain policy is attached to this VM. Let’s now observe the physical disk placement:
As you can see: each ESXi host (11 en 12) are hosting 2 component + 1 witness object. On top of that there’s a witness object running on the vSAN witness. Available data is copied four times, so each GB of space on the VM guest OS level will cost you 4 GB of space on the vSAN level (2 GB per ESXi host).
The test VM uses about 14,7 GB on the vSAN level, this is about 4 times higher than the actual consumption on the GuestOS level (a little less than 4 GB). You can also see the allocated disk space is 65,5 GB, this is about 4 times higher than the size (16 GB) of the virtual hard disk for this VM.
Emulate a host and diskgroup failure
Now let’s test the Nested Fault Domain feature. The VM is currently running on esxi11.viktorious.local, let’s first switch this ESXi server off and see if HA kicks in. Within about 2 minutes the VM is up and running again and responding to network requests. Of course some of the vSAN components are missing because of this failure:
The next test is to remove a capacity disk in one of the diskgroups on the remaining ESXi host and see what happens:
As you can see some of the component are reported missing because of an absent vSAN disk. The VM continues to run and is still available, despite some serious issues at the vSAN level. After switching on host esxi11.viktorious.local again, we see things are getting back to normal:
After replacing my failed disk, marking the new disk as a capacity disk and adding it to the failed diskgroup vSAN will have to do some resycing work. After the resyncing completes things are back to normal:
I think this shows the added value of Nested Fault Domains for 2-node clusters and how it provides a higher availability option in 2-node vSAN cluster setup. Feel free to leave a comment below.
