In more severe failures, entire loss of the underlying storage system or SAN can occur, leading to ESXi hosts continuing to run, but with no storage access. This is a different type of failure than a host failure—as the host is still online but it cannot continue to run its VMs as there are no available paths to the local FlashArray.
Depending on the failure and on the configuration of the stretched cluster (uniform or non-uniform), vSphere reacts differently and the failover mechanism changes.
Storage Failure Response with a Non-Uniform Stretched Cluster
As discussed previously in this guide, a non-uniform stretched cluster is a cluster of ESXi hosts which are split across to physical sites, usually half in one datacenter and half in another datacenter. Each datacenter has a FlashArray and those two FlashArrays have ActiveCluster enabled on one or more volumes allowing the storage volume(s) to be presented simultaneously at both sites. The non-uniform portion of this configuration defines what paths to the storage the hosts see. In a non-uniform configuration, hosts only have paths to the volume (or volumes) via the FlashArray that is local to their datacenter.
In other words, if the FlashArray becomes unavailable, the ESXi hosts local to it no longer have access to the storage and a vSphere HA failover must occur to reboot any affected VMs on the remote hosts in the cluster that have access to the storage via the remote FlashArray.
In this scenario, essentially any storage-related failure will cause a vSphere HA failover such as:
- Loss of local FlashArray
- Accidental or purposeful removal of volume from access to hosts by administrator
- Loss of SAN connectivity due to failure, power loss or administrative change
- Failure of host bus adapters (HBAs) in host or connecting cables
These failures all lead to the same result in the presence of non-uniform configurations: loss of storage connectivity of the host and a vSphere HA failover.
In the following environment, there are eight hosts total, four are in site “A” and four are in site “B”. There is a VMFS datastore presented to all hosts in the cluster, with a total of 8 VMs on it.
The datastore is hosted on a FlashArray volume that has been stretched to both FlashArrays using ActiveCluster.
Since this is a non-uniform configuration, the “A” hosts only have access to the volume via paths to the “A” FlashArray, and the “B” hosts only have access to the volume via paths to the “B” FlashArray.
In this cluster, half of the 8 virtual machines are running on “B” hosts and half are running on “A” hosts, so a storage failure on either side will cause 4 VMs to be restarted on the remaining hosts on the other side.
For this example, FlashArray “B” will experience a failure:
This causes the VMFS datastore to go inaccessible on the “B” hosts.
But because it is presented through both FlashArrays via ActiveCluster, the “A” hosts continue to have access to the datastore:
Once the timeouts have been reached (dependent on the APD/PDL timeout configuration) if APD/PDL responses have been enabled, ESXi will shut down the VMs on the hosts that lost storage access and restart them on hosts with surviving storage connections to the volume. The VMs will be shut down, which makes them briefly marked as inaccessible.
The VMs will then come back online and be powered-on elsewhere. When the FlashArray comes back online, the storage can then be seen again by the “A” hosts. If there are host-affinity rules, the VMs will be moved back almost immediately by vSphere DRS. Otherwise, they will remain where they are until manually moved, another failure occurs, or resource usage demands vSphere DRS rebalance the VMs across the cluster.
Storage Failure Response with a Uniform Stretched Cluster
A uniform stretched cluster means that all hosts in the cluster have paths to a stretched volume through both FlashArrays servicing that volume via ActiveCluster. This configuration provides an additional level of resilience for virtual machines, as VMs can continue to run non-disruptively through the failure of an entire FlashArray, or the connectivity to it.
In the case of all hosts in one site losing all storage access where those hosts cannot access their local FlashArray, nor their remote one, the failover process is identical to the process shown in the previous section on non-uniform failover. In that case, vSphere HA will restart the affected virtual machines on the hosts in the remote site.
For uniform configurations, the case where just a single array fails is different. In this case, VMs can continue to run on their host, as those hosts can still access the storage, but via paths to the remote FlashArray. Therefore, this is not a case of a vSphere HA restart,which has downtime until the VM can be rebooted, but it is a case of multipathing simply failing over to the paths to the remote FlashArray. A multipathing failover is entirely non-disruptive, as a VM reboot is not required.
The below environment is configured in a uniform fashion, so all eight hosts have access to the VMFS protected via ActiveCluster through both FlashArrays:
Furthermore, the preferred FlashArray is set on the host object, so that only the paths for a given host to its local FlashArray are in-use (denoted by the (I/O)).
This ensures that, while the paths are available, reads and writes go down optimized paths to provide the best possible performance. The non-optimized paths are paths to the VMFS via the remote FlashArray (non-optimized) and will only be used in absence of any optimized paths.
After a failure of a FlashArray, the datastores from that array that are not protected by ActiveCluster go offline:
The ActiveCluster-enabled volume “vMSC-VMware::nelson-stretched-volume” stays online.
The paths to that volume to the failed FlashArray are gone, but the paths to the remote site remain available and become active.
At this point, the virtual machines are running on non-optimized paths, meaning that their I/Os are running the WAN to the remote FlashArray, which will incur greater latency than if the VMs were running on hosts local to their FlashArray. If the FlashArray failure is expected to be extended, it might be advisable to vMotion the VMs running on non-optimized paths to hosts in the other datacenter that have paths to their local FlashArray available. If the failed FlashArray is expected to be recovered soon (like in the case of a temporary loss of power), the simplest option may be to just leave the VMs where they are. They will resume running on optimized paths as soon as the FlashArray comes back online.