Impact to Data Path

User Guides for VMware Solutions
Audience
Public
Content Type
User Guides
Source Type
Documentation

Here we will review data path impact situations.

There are 2 main types of possible events that lead to an impact to the connection of the ESXi Hosts to the FlashArray protocol-endpoint and subsidiary LUNs.

Storage Network Disruption

Such as a switch issue, networking problem, or ports failing in the SAN.

Protocol Endpoint Disconnection

Either from disconnecting the PE from the FlashArray Host Group or Host, or from a APD or PDL event to the PE.

Example of Data Path Impact:

The observed impact to the Data Path will be similar if either: all active data paths go down because of a switch or networking issue, or, by removing the protocol endpoint from the host. Network connectivity issues will result in an APD event, while the device backing being disconnected will lead to a PDL event. In both events, the connectivity for both the Protocol Endpoint and all subsidiary LUNs will be lost. The VMs will be in a hung or failed state and will be inaccessible, and the vVol Datastore will report as inaccessible.

Recall when connectivity to the management path was lost there was no impact to running vVol VMs. This was because the Data Path was not impacted during those events. The following example will have the opposite event-- where there is impact to the Data Path and no impact to the Management Path. We'll see that the VASA Provider will stay online and healthy, but the connectivity to the Protocol Endpoint will be lost, which in turn leads to impact to the vVol VMs and a true outage.

In the example that follows, the network interfaces on the FlashArray will be disabled and the impact from that loss will be shown.

Before disabling the iSCSI interfaces on the FlashArray, here is the healthy environment. There is active I/O going to the FlashArray.

The Storage Container, Protocol Endpoint and VASA Provider are all showing up as healthy:


[root@ESXi-6:~] esxcli storage vvol storagecontainer list
sn1-m20-c08-17-vvol-container
   StorageContainer Name: sn1-m20-c08-17-vvol-container
   UUID: vvol:79eb90a400c632c4-a47630fae8da5412
   Array: com.purestorage:801c80a2-dd56-4d38-86a2-c6fabc38cc91
   Size(MB): 8589934592
   Free(MB): 8589442761
   Accessible: true
   Default Policy:
   
[root@ESXi-6:~] esxcli storage vvol protocolendpoint list
naa.624a9370801c80a2dd564d380001104b
   Host Id: naa.624a9370801c80a2dd564d380001104b
   Array Id: com.purestorage:801c80a2-dd56-4d38-86a2-c6fabc38cc91
   Type: SCSI
   Accessible: true
   Configured: true
   Lun Id: naa.624a9370801c80a2dd564d380001104b
   Remote Host:
   Remote Share:
   NFS4x Transport IPs:
   Server Scope:
   Server Major:
   Auth:
   User:
   Storage Containers: 79eb90a4-00c6-32c4-a476-30fae8da5412
   
[root@ESXi-6:~] esxcli storage vvol vasaprovider list
sn1-m20-c08-17-ct0
   VP Name: sn1-m20-c08-17-ct0
   URL: https://10.21.203.31:8084/version.xml
   Status: online
   Arrays:
         Array Id: com.purestorage:801c80a2-dd56-4d38-86a2-c6fabc38cc91
         Is Active: true
         Priority: 200

The host connectivity looks good from vCenter as well.

Here is how the FlashArray iSCSI interfaces are being disabled:


     root@sn1-m20-c08-17-ct0:~# purenetwork disable ct0.eth6 ct0.eth7 ct1.eth6 ct1.eth7
Name      Enabled  Subnet  Address       Mask           Gateway      MTU   MAC                Speed       Services  Subinterfaces
ct0.eth6  False    -       10.21.203.33  255.255.255.0  10.21.203.1  1500  90:e2:ba:2c:90:71  10.00 Gb/s  iscsi     -
ct0.eth7  False    -       10.21.203.34  255.255.255.0  10.21.203.1  1500  90:e2:ba:2c:90:70  10.00 Gb/s  iscsi     -
ct1.eth6  False    -       10.21.203.35  255.255.255.0  10.21.203.1  1500  90:e2:ba:7f:bc:3d  10.00 Gb/s  iscsi     -
ct1.eth7  False    -       10.21.203.36  255.255.255.0  10.21.203.1  1500  90:e2:ba:7f:bc:3c  10.00 Gb/s  iscsi     -

Now that the interfaces are disabled, let's see what the impact looks like.

From the vCenter UI you will see that the vVol Datastore is inaccessible and the hosts are not connected to the vVol Datastore:

From the FlashArray GUI we can see that the workload stopped when the networking was lost (as would be expected).

From the esxcli we can see that the VASA Provider is still in a healthy state, but the storage container and protocol endpoint are inaccessible now.


[root@ESXi-6:~] esxcli storage vvol storagecontainer list
sn1-m20-c08-17-vvol-container
   StorageContainer Name: sn1-m20-c08-17-vvol-container
   UUID: vvol:79eb90a400c632c4-a47630fae8da5412
   Array: com.purestorage:801c80a2-dd56-4d38-86a2-c6fabc38cc91
   Size(MB): 8589934592
   Free(MB): 8589447423
   Accessible: false
   Default Policy:
   
[root@ESXi-6:~] esxcli storage vvol protocolendpoint list
naa.624a9370801c80a2dd564d380001104b
   Host Id: naa.624a9370801c80a2dd564d380001104b
   Array Id: com.purestorage:801c80a2-dd56-4d38-86a2-c6fabc38cc91
   Type: SCSI
   Accessible: false
   Configured: true
   Lun Id: naa.624a9370801c80a2dd564d380001104b
   Remote Host:
   Remote Share:
   NFS4x Transport IPs:
   Server Scope:
   Server Major:
   Auth:
   User:
   Storage Containers: 79eb90a4-00c6-32c4-a476-30fae8da5412
   
[root@ESXi-6:~] esxcli storage vvol vasaprovider list
sn1-m20-c08-17-ct0
   VP Name: sn1-m20-c08-17-ct0
   URL: https://10.21.203.31:8084/version.xml
   Status: online
   Arrays:
         Array Id: com.purestorage:801c80a2-dd56-4d38-86a2-c6fabc38cc91
         Is Active: true
         Priority: 200

Here is a look at the logging from the ESXi host's vmkernel and vvold log that are of note:


## vmkernel log ##
2019-02-27T23:15:53.177Z cpu37:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:7 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2019-02-27T23:15:53.177Z cpu37:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000008 TARGET: iqn.2010-06.com.purestorage:flasharray.107f4b1e374f4d6b TPGT: 1 TSIH: 0]
2019-02-27T23:15:53.177Z cpu37:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.21.203.188:32535 R: 10.21.203.36:3260]
2019-02-27T23:15:53.181Z cpu37:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:6 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2019-02-27T23:15:53.181Z cpu37:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000007 TARGET: iqn.2010-06.com.purestorage:flasharray.107f4b1e374f4d6b TPGT: 1 TSIH: 0]
2019-02-27T23:15:53.181Z cpu37:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.21.203.163:20319 R: 10.21.203.36:3260]
2019-02-27T23:16:02.287Z cpu31:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2019-02-27T23:16:02.287Z cpu31:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000002 TARGET: iqn.2010-06.com.purestorage:flasharray.107f4b1e374f4d6b TPGT: 1 TSIH: 0]
2019-02-27T23:16:02.287Z cpu31:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.21.203.188:21959 R: 10.21.203.33:3260]
2019-02-27T23:16:02.291Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2019-02-27T23:16:02.291Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000001 TARGET: iqn.2010-06.com.purestorage:flasharray.107f4b1e374f4d6b TPGT: 1 TSIH: 0]
2019-02-27T23:16:02.291Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.21.203.163:62519 R: 10.21.203.33:3260]
2019-02-27T23:16:04.569Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:3 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2019-02-27T23:16:04.569Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000004 TARGET: iqn.2010-06.com.purestorage:flasharray.107f4b1e374f4d6b TPGT: 1 TSIH: 0]
2019-02-27T23:16:04.569Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.21.203.188:57333 R: 10.21.203.34:3260]
2019-02-27T23:16:05.332Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:5 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2019-02-27T23:16:05.332Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000006 TARGET: iqn.2010-06.com.purestorage:flasharray.107f4b1e374f4d6b TPGT: 1 TSIH: 0]
2019-02-27T23:16:05.332Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.21.203.188:62535 R: 10.21.203.35:3260]
2019-02-27T23:16:05.337Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:4 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2019-02-27T23:16:05.337Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000005 TARGET: iqn.2010-06.com.purestorage:flasharray.107f4b1e374f4d6b TPGT: 1 TSIH: 0]
2019-02-27T23:16:05.337Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.21.203.163:25098 R: 10.21.203.35:3260]
2019-02-27T23:16:05.340Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:2 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2019-02-27T23:16:05.340Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: 00023d000003 TARGET: iqn.2010-06.com.purestorage:flasharray.107f4b1e374f4d6b TPGT: 1 TSIH: 0]
2019-02-27T23:16:05.340Z cpu32:4994511)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 10.21.203.163:26364 R: 10.21.203.34:3260]
2019-02-27T23:16:05.345Z cpu11:2097313)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1844: Could not select path for device "naa.624a9370801c80a2dd564d380001104b".
2019-02-27T23:16:05.345Z cpu11:2097313)ScsiDevice: 8707: No Handlers registered! (naa.624a9370801c80a2dd564d380001104b)!
2019-02-27T23:16:05.345Z cpu11:2097313)ScsiDevice: 5978: Device state of naa.624a9370801c80a2dd564d380001104b set to APD_START; token num:1
2019-02-27T23:16:05.345Z cpu11:2097313)StorageApdHandler: 1203: APD start for 0x43055b4d1f20 [naa.624a9370801c80a2dd564d380001104b]
2019-02-27T23:16:05.345Z cpu1:2097602)StorageApdHandler: 419: APD start event for 0x43055b4d1f20 [naa.624a9370801c80a2dd564d380001104b]
2019-02-27T23:16:05.345Z cpu1:2097602)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.624a9370801c80a2dd564d380001104b] has entered the All Paths Down state.
2019-02-27T23:16:07.188Z cpu28:2097313)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1844: Could not select path for device "naa.624a9370801c80a2dd564d380001104b".
2019-02-27T23:16:12.290Z cpu28:2097313)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1844: Could not select path for device "naa.624a9370801c80a2dd564d380001104b".
2019-02-27T23:16:12.294Z cpu28:2097313)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1844: Could not select path for device "naa.624a9370801c80a2dd564d380001104b".
2019-02-27T23:16:14.572Z cpu28:2097313)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1844: Could not select path for device "naa.624a9370801c80a2dd564d380001104b".
2019-02-27T23:16:15.335Z cpu28:2097313)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1844: Could not select path for device "naa.624a9370801c80a2dd564d380001104b".
2019-02-27T23:16:15.339Z cpu28:2097313)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1844: Could not select path for device "naa.624a9370801c80a2dd564d380001104b".
2019-02-27T23:16:15.343Z cpu28:2097313)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1844: Could not select path for device "naa.624a9370801c80a2dd564d380001104b".
2019-02-27T23:18:25.349Z cpu1:2097602)ScsiDevice: 8707: No Handlers registered! (naa.624a9370801c80a2dd564d380001104b)!
2019-02-27T23:18:25.349Z cpu1:2097602)StorageApdHandler: 609: APD timeout event for 0x43055b4d1f20 [naa.624a9370801c80a2dd564d380001104b]
2019-02-27T23:18:25.349Z cpu1:2097602)StorageApdHandlerEv: 126: Device or filesystem with identifier [naa.624a9370801c80a2dd564d380001104b] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will n$

## vvold log ##
2019-02-27T23:16:07.282Z info vvold[7746429] [Originator@6876 sub=Libs] LunImpl: IsAccessible = false, path state = 2, lun state 2
2019-02-27T23:16:07.282Z info vvold[7746429] [Originator@6876 sub=Default] HostManager::AddOrUpdatePE PE lookup (
--> SCSI PE, ID (host: naa.624a9370801c80a2dd564d380001104b, vasa: naa.624a9370801c80a2dd564d380001104b) (inaccessible, configured)
-->  Lun Id: naa.624a9370801c80a2dd564d380001104b)
2019-02-27T23:16:07.282Z info vvold[7746429] [Originator@6876 sub=Default] ProtocolEndpoint::InitializeFromInbandInfo Initializing SCSI PE (naa.624a9370801c80a2dd564d380001104b) using inband info
2019-02-27T23:16:07.282Z info vvold[7746429] [Originator@6876 sub=Default] HostManager::DiscoverSCSIPEs peVec.size=2, _peMap.size=2, setPEContextScheduled=false
2019-02-27T23:16:07.282Z info vvold[7746429] [Originator@6876 sub=Default] HostManager::RefreshPEMap PE (naa.624a9370801c80a2dd564d380001104b, naa.624a9370801c80a2dd564d380001104b) discovered inband, is inaccessible, will skip from setPEContext!
...
2019-02-27T23:16:07.300Z info vvold[7746429] [Originator@6876 sub=Default] VendorProviderMgr::SetPEContext: SCSI PE (uniqueIdentifier=naa.624a93702dcf29ad6aca491300089aff lunId=naa.624a93702dcf29ad6aca491300089aff ipAddress=__none__ serverMount=__none__ serverScope=__none__ serverMajor=__none__ authType=__none__ transportIpAddress=)
2019-02-27T23:16:07.300Z info vvold[7746429] [Originator@6876 sub=Default] HostManager::GetContainerPEAccessibility arrayId: com.purestorage:801c80a2-dd56-4d38-86a2-c6fabc38cc91, cid: 79eb90a4-00c6-32c4-a476-30fae8da5412, accessible: 0,checkAllPEsAccessibility: false containerType: SCSI, APD: 1
2019-02-27T23:16:07.300Z info vvold[7746429] [Originator@6876 sub=Default] StorageContainer::UpdateAccessibility: Accessibility of cId vvol:79eb90a400c632c4-a47630fae8da5412 (name: sn1-m20-c08-17-vvol-container) changed (Accessible -> Unaccessible), hostd was notified (VP: accessible)

Essentially here we have an APD event for all vVol related objects to the FlashArray.

After enabling the iSCSI interfaces on the FlashArray, the vVols Datastore and host connectivity recovered automatically on its own.