When an event impacts the existing connection to the Management Path, there is no disruption to the I/O coming from any currently running vVol VMs. The disruption here comes into play with the communication via VASA between the ESXi Hosts, vCenter and the Storage Array. This disruption will prevent powered off vVol VMs from being powered on, vSphere vMotion(s) and Storage vMotion(s) will fail, vVol VMs won't be able to have their settings edited or updated, no new VMs can be created on the vVol Datastore, and the vVol Datastore will show as inaccessible in the vCenter UI. Essentially no reconfigurations will be possible until the management path comes back online. Examples include:
- Creating new virtual disks.
- Resizing virtual disks.
- Deleting virtual disks.
- Assigning a storage policy to a VM or virtual disk.
- Powering off a VM.
- Powering on a VM.
- Moving a VM.
However, it is important to understand, that the existing vVol VMs that are powered on will continue to stay powered-on and service I/O. As an example, in a non-vVol environment, what happens to you I/O if you cannot access the FlashArray web management interface? It of course continues to run, but you cannot make any changes to the storage configuration until you regain access. This is identical in concept to vSphere losing management path access--vSphere cannot make any configuration changes until VASA access is restored. If management path (VASA) access goes down: DO NOT PANIC and start rebooting VMs--instead log into a running VM to make sure the data path is not also down, then start troubleshooting why vSphere cannot access VASA.
Just a note before starting, there is no impact to running vVol VMs when re-registering storage providers that are either active or offline. In a video example towards the bottom of this page, the Storage Providers are in an offline state when they are re-registered. This would include if the storage providers are removed when there are active vVol VMs running on that vCenter and then are registered again. Removing and registering storage providers in not impactful to the vVol Data Path.
Let's cover some possible events that lead to an impact on the connection of the ESXi Hosts or vCenter Server to the FlashArray VASA Providers. Then we'll review what this impact can look like.
Firewall or Security Rules
An example of this is port 8084 being blocked or restricted between the vCenter Server and ESXi Host management network and the FlashArray's management network for CT0 and CT1.
Here is an example of a firewall change after the storage providers have been registered and vVols is currently in use.
|
Here we have the Storage Providers Online and Active:
|
|---|
|
Then looking at the esxcli, we can see the storage container is online and in sync. Using nc -z we can see that the ESXi host can reach CT0 and CT1 over port 8084. |
After showing we have a good and healthy configuration, let's say someone made a firewall change or blocked port 8084 on an upstream switch....
|
Okay, now we can see that the vVol Datastore is inaccessible and the Storage Providers are offline:
|
|
When checking on the esxcli we can see the vasaprovider is in 'syncError' and the Storage Container is inaccessible. Using nc -z notice that the ESXi host is unable to connect to CT0 or CT1 over port 8084, but ping is still successful. Note that the protocol endpoint is still active and accessible, as the Data Path has not been impacted. In the vCenter Server's /var/log/vmware/vmware-sps/sps.log you will find an error that says 'Connection reset' and 'Connection refused' as well: |
Confirming the above, we would then need to fix the firewall rules. After the firewall rules were fixed, the Storage Providers came back online and the vVold Datastore was accessible again. An important thing to point out here is that in such a situation, the VMs stay online and continue to push workload. Any operation to the vVol VM would fail and any powered off vVol VMs would fail to power on.
Certificate Issues
These can be with the vCenter Server, ESXi Hosts or FlashArray. Examples of probable issues include expired certificates, outdated certificate information, accidently registering a different vCenter with the FlashArray's VASA Provider, or ESXi hosts unable to sync with the vCenter Storage Provider Certificate.
Here's an example where we have a working vCenter Server become impacted by another non-linked vCenter Server registering the FlashArray's VASA Provider. This would cause the existing vCenter that is registered with the FlashArray to have invalid certificates.
|
Here we have healthy Storage Providers:
After another vCenter has been registered with the Storage Providers they are no longer online and healthy:
From esxcli we can see that the storage container is offline and the VASA provider is out of sync: The interesting log lines from the ESXi hosts /var/log/vvold and the vCenter Server's /var/log/vmware/vmware-sps/sps.log are as follows: The only way to correct this is to re-register the Storage Providers for the vCenter Server that is supposed to have access to vVols on the FlashArray. Once the Storage Providers are re-registered then everything comes back online and is healthy again. |
Prior to Purity 5.3 and VASA 1.1.0, the FlashArray's VASA Provider Certificates are stored locally on the FlashArray controllers. This requires the controllers Storage Provider being re-registered when a controller is replaced or is upgraded. There have been instances where the storage providers were not re-registered post NDU (non-disruptive upgrade) or controller replacement. Everpure is working to configure the VASA Providers in a method that would persist during controller replacements and upgrades.
Beginning with Purity 5.3 and VASA 1.1.0, the FlashArray's VASA Provider Certificates are stored within Purity and managed with the purecert API. Granted, during controller replacements and Hardware NDU's the storage providers will still need to be re-registered. This is do to the trusted root certs from VMCA that are sent to VASA are not persisted between Hardware changes. This is being corrected for future releases of Purity and VASA.
The following example will outline what happens when the controllers are not re-registered following a hardware NDU.
|
Here are the healthy storage providers:
Then both controllers are replaced/upgraded. We can see that both providers are now offline:
Then looking at the vCenter Server's sps log, we can see the cert failures: Then looking at the ESXi Host's vvold log, we can see the authentication and cert failures: This is essentially the same from a symptom perspective as the previous example showed. The resolution is the same as well; re-register the storage providers and everything comes back online and healthy. |
Changes to the Storage Provider Account
Examples of this are password changes, permission changes, or account deletion.
When testing these different scenarios Pure found that when changing the password or the user permission levels, there was no effect on the existing storage providers and they did not go offline. The explanation behind this behavior is due to the username/password combination being used when initially registering the storage providers; once that connection is made to the FlashArray's VASA Provider and the ESXi hosts have authenticated, there is no request to validate the password again. This is due to the certificate and thumbprints being valid between the vCenter Server, ESXi Hosts and the FlashArray VASA Provider. Any future attempts to register the storage provider with additional vCenters will fail if using the incorrect password.
If either an AD User or FlashArray Local User that was used to register the storage provider is deleted, impact to the Management Path will be observed. The Storage Provider will be offline and the ESXi hosts will be in a sync error state with the VASA Provider. There are two solutions to this issue: either re-create the deleted FlashArray/AD User or delete the registered storage providers in vCenter and then re-register each controllers storage provider with a new user.
Here is an example of what can be seen if the user that the storage provider was registered with is deleted or otherwise removed.
|
In vCenter you will see that the Active Storage Provider is offline and that the standby is unable to takeover for the failed provider:
|
|---|
|
This is an error you'll see in the vCenter Server's /var/log/vmware/vmware-sps/sps.log: |
After adding back the user, just refresh the Storage Providers, rescan the FlashArray Storage Provider, and it'll be back online.
Management Network Outage
Let's say an ESXi host of the vCenter Server loses access to the management network. The impact and what the impact looks like will be very similar to what is observed on firewall changes or issues.
FlashArray VASA Service Connectivity
Connectivity issues can be seen if there is a FlashArray Controller replacement, failure or reboot (such as during a Purity Upgrade), if the service is unreachable on one or both controllers, or the service is stopped for any reason.
|
Should just one of the services fail, you will see it updated in the vCenter Server UI as offline. The standby provider will take over as active should the failed provider have been the active provider: |
|---|
|
|
|
Should both of the services be disrupted, then both storage providers will show up as offline in the vCenter Server UI: |
|---|
|
In addition to the providers showing offline in the UI, in the /var/log/vmware/vmware-sps/sps.log there will be messages of Bad Gateway: |
vCenter Server is Down or Isolated
In this example the vCenter Server is going to be powered off. The expected outcome would be that any vCenter related operations would not work, such as applying storage policies, taking managed snapshots, or a vSphere vMotion between hosts. However, any vVol related operations that the host can run should still continue to work, for example powering on a vVol VM or editing the VMs settings.
|
vCenter Failure Example |
|---|
|
Let's first check that the vVol container and VASA provider are in a good state. As we can see below, both are in a good state. Next is to power off the vCenter (for the purposes of this example)
Now that the vCenter Server is off, let's quickly check that the storage container and VASA provider status. Okay, both the storage container and VASA provider are still in a good state. |
|
Next we'll power on a vVol VM that's powered off. Then edit a vVol VM's settings to add an additional VMDK.
|
From the example above we can see that even though the vCenter is down, the ESXi host is still able to communicate with the VASA Provider on the array. Power on operations worked as did adding a VMDK to another VM.
Why the vCenter Server should not run on vVols
In the previous example, we saw that even if vCenter was down the ESXi host could still issue management requests. However, what would have happened if something was changed on the FlashArray or with how the ESXi hosts authenticated with the VASA Provider? For example, while the vCenter Server was powered off, the user that registered the storage providers had the password changed? Another example would be if the IP address on CT0.ETH0 or CT1.ETH1, and another example would be if the DNS server went down and the Storage Providers were registered with the FQDN. There are some other examples, but these are some of the more common issues that have been observed by Pure Support. In the event that any of these events occurred while vCenter Server was down, this would cause the ESXi host to fail authentication to the VASA Provider. That authentication could not be re-authorized until the vCenter Server was back online and the Storage Providers are re-registered.
When the vCenter Server is on a VMFS Datastore, then that's an easy ask to just power back on the vCenter Server. However, what happens if the vCenter Server is on the vVol Datastore that just had something change and the ESXi host is no longer able to authenticate with VASA? Any management path related request would fail. Powering on a VM is a management path request. What ends up happening is that the vCenter Server is now unable to power on, which would mean that the Storage Providers can not be re-registered. This is a circular dependency type of issue and thus why we strongly recommend not to run the vCenter Server on vVols. Similar points could be made about a DNS Server or AD Server as well, but if the environment is looking at DNS and AD as crucial services they should be in an HA setup and if one VM was down, it should not take down all of AD and DNS. Should that be the case, Everpure would recommend getting the AD and DNS services in an HA setup that does not have a singular point of failure (storage, network, compute, etc).
|
Here is a video demo of a failure in the management path and what the impact can be. |
|---|
|
|