NVMe-FC ESXi Host Configuration

Note:

Please ensure the nfnic version is 5.0.0.41 or greater because of a purple screen possibility in ESXi when using UCS with NVMe-FC and ESXi.

The first configuration step is to enable NVMe-oF on your ESXi host(s). In this environment, I have a cluster that has two hosts, each with supported NVMe-FC HBAs:

With some earlier NVMe HBA drivers, when you click on Storage Adapters, there isn't a listing for controllers, namespaces, etc. This is because the NVMe "feature" driver is not enabled by default.

Note:

With newer versions of certain adapters, the NVMe feature is enabled by default, such as with Emulex cards with version 12.8 or later. So, in that case the following step is not necessary.

For versions that do not have it enabled by default, this change cannot be done in the vSphere Client, but instead only through cmd line or a tool that can make esxcli calls such as PowerCLI.

SSH into the host(s).

Note:

Ensure that your installed HBA driver version supports NVMe-FC by checking the VMware Compatability Guide for IO Device with Features: NVMe/FC. If the currently installed driver does not have this listed, you will need to install a version that does.

Run the following command to enable this feature:

Emulex:


esxcli system module parameters set -m lpfc -p "lpfc_enable_fc4_type=3"

Qlogic:


esxcfg-module -s "ql2xnvmesupport=1" qlnativefc

Then reboot the host.
Repeat this on each host in the cluster.
Once the host has come back up, navigate back to the storage adapters. You will see new adapters added (one for each port), which will likely be something like vmhba65 and so on.
Once you click on one, you will see more information appear in the details panel:
If your zoning is complete at this point (this is not covered here–but the process is the same with SCSI-FC) the WWNs are the same for the host and array.
1. So if zoning was already done for FC-SCSI, no additional steps are needed.
2. If zoning is not complete, follow the normal process you use for SCSI zoning.
3. The controllers will automatically appear. Unlike RoCE, you do not need to manually configure the controllers because they will auto-discover.
4. If you do not see anything listed here, ensure zoning is complete and upgrade to the latest HBA drivers.

You can see this event in the vmkernel log on the ESXi host (/var/log/).


2021-01-16T01:29:10.112Z cpu22:2098492)NVMFDEV:302 Adding new controller nqn.2010-06.com.purestorage:flasharray.6d122f70cac7d785 to active list
 2021-01-16T01:29:10.112Z cpu22:2098492)NVMEDEV:2895 disabling controller…
 2021-01-16T01:29:10.112Z cpu22:2098492)NVMEDEV:2904 enabling controller…
 2021-01-16T01:29:10.112Z cpu22:2098492)NVMEDEV:877 Ctlr 257, queue 0, update queue size to 31
 2021-01-16T01:29:10.112Z cpu22:2098492)NVMEDEV:2912 reading version register…
 2021-01-16T01:29:10.112Z cpu22:2098492)NVMEDEV:2926 get controller identify data…
 2021-01-16T01:29:10.112Z cpu22:2098492)NVMEDEV:5101 Controller 257, name nqn.2010-06.com.purestorage:flasharray.6d122f70cac7d785#vmhba65#524a93775676b304:524a93775676b304
 2021-01-16T01:29:10.112Z cpu22:2098492)NVMEDEV:4991 Ctlr(257) nqn.2010-06.com.purestorage:flasharray.6d122f70cac7d785#vmhba65#524a93775676b304:524a93775676b304 got slot 0
 2021-01-16T01:29:10.112Z cpu22:2098492)NVMEDEV:6156 Succeeded to create recovery world for controller 257

For optimal performance with Pure backed devices, a claimrule needs to be added on the ESXi hosts that are going to be used with NVMe-FC. Please note that there are two different versions of this claimrule depending on the ESXi version.
1. On ESXi 8.0 U1 and later, please run the following two commands. Please note no reboot is required:
```
esxcli storage core claimrule add --rule 102 -t vendor --nvme-controller-model "Pure*" -P HPP -g "pss=LB-Latency,latency-eval-time=180000,sampling-ios-per-path=16"

esxcli storage core claimrule load
```
2. On ESXi 8.0 GA and earlier, please run the following two commands. (more information here) Please note no reboot is required:
```
esxcli storage core claimrule add --rule 102 -t vendor -P HPP -V NVMe -M "Pure*" --config-string "pss=LB-Latency,latency-eval-time=180000"

esxcli storage core claimrule load
```
The next step is to create the host object on the FlashArray. In NVMe-oF, initiators use something called an NVMe Qualified Name (NQN).

The initiator has one and so does the target (the FlashArray). With NVMe-FC, NQNs do not replace FC WWNs–they both exist.

The WWN of each side is what is advertised on the FC layer to enable physical connectivity and zoning. The NQN is what enables the NVMe layer to communicate to the correct endpoints on the FC fabric. You can look at it in a similar way as networking in IP (MAC addresses and IPs).
Connect at the physical layer (WWN zoning),
then the logical layer (NQNs).

For each ESXi host, you need to create a host object on the FlashArray, then add the NQN to it. So where do you get the NQN? However, not from the vSphere Client. For now, you need to use esxcli.
So, SSH back into the ESXi host and run:
```
esxcli nvme info get
```
Copy the NQN.
Then log into the FlashArray.