Virtual Disk Choice

Storage provisioning in virtual infrastructure involves multiple steps of crucial decisions. VMware vSphere offers three virtual disks formats: thin, zeroedthick and eagerzeroedthick.

1. Thin—thin virtual disks only allocate what is used by the guest. Upon creation, thin virtual disks only consume one block of space. As the guest writes data, new blocks are allocated on VMFS, then zereod out, then the data is committed to storage. Therefore there is some additional latency for new write
2. Zeroedthick (lazy)— zeroed thick virtual disks allocate all of the space on the VMFS upon creation. As soon as the guest writes to a specific block for the first time in the virtual disk, the block is first zeroed, then the data is committed. Therefore there is some additional latency for new writes. Though less than thin (since it only has to zero—not also allocate), there is a negligible performance impact between zeroedthick (lazy) and thin.
3. Eagerzeroedthick—eagerzeroedthick virtual disks allocate all of their provisioned size upon creation and also zero out the entire capacity upon creation. This type of disk cannot be used until the zeroing is complete. Eagerzeroedthick has zero first-write latency penalty because allocation and zeroing is done in advance, and not on-demand.

   Prior to WRITE SAME support, the performance differences between these virtual disk allocation mechanisms were distinct. This was due to the fact that before an unallocated block could be written to, zeroes would have to be written first causing an allocate-on-first-write penalty (increased latency). Therefore, for every new block written, there were actually two writes; the zeroes then the actual data. For thin and zeroedthick virtual disks, this zeroing was on-demand so the penalty was seen by applications. For eagerzeroedthick, it was noticed during deployment because the entire virtual disk had to be zeroed prior to use. This zeroing caused unnecessary I/O on the SAN fabric, subtracting available bandwidth from “real” I/O.

   To resolve this issue, VMware introduced WRITE SAME support. WRITE SAME is a SCSI command that tells a target device (or array) to write a pattern (in this case, zeros) to a target location. ESXi utilizes this command to avoid having to actually send a payload of zeros but instead simply communicates to any array that it needs to write zeros to a certain location on a certain device. This not only reduces traffic on the SAN fabric, but also speeds up the overall process since the zeros do not have to traverse the data path.

   This process is optimized even further on the Everpure FlashArray. Since the array does not store space-wasting patterns like contiguous zeros on the array, the zeros are discarded and any subsequent reads will result in the array returning zeros to the host. This additional array-side optimization further reduces the time and penalty caused by pre-zeroing of newly-allocated blocks.

   With this knowledge, choosing a virtual disk is a factor of a few different variables that need to be evaluated. In general, Everpure makes the following recommendations:

4. Lead with thin virtual disks. They offer the greatest flexibility and functionality and the performance difference is only at issue with the most sensitive of applications.
5. For highly-sensitive applications with high performance requirements, eagerzeroedthick is the best choice. It is always the best-performing virtual disk type.
6. In no situation does Pure Storage recommend the use of zeroedthick (thick provision lazy zeroed) virtual disks. There is very little advantage to this format over the others and can also lead to stranded space as described in this post.

   With that being said, for more details on how these recommendations were decided upon, refer to the following considerations. Note that at the end of each consideration is a recommendation but that recommendation is valid only when only that specific consideration is important. When choosing a virtual disk type, take into account your virtual machine business requirements and utilize these requirements to motivate your design decisions. Based on those decisions, choose the virtual disk type that is best suitable for your virtual machine.

7. Performance—with the introduction of WRITE SAME (more information on WRITE SAME can be found in the section Block Zero or WRITE SAME) support, the performance difference between the different types of virtual disks is dramatically reduced—almost eliminated. In lab experiments, a difference can be observed during writes to unallocated portions of a thin or zeroedthick virtual disk. This difference is negligible but of course still non-zero. Therefore, performance is no longer an overridingly important factor in the type of virtual disk to use as the disparity is diminished, but for the most latency-sensitive of applications eagerzeroedthick will always be slightly better than the others. Recommendation: eagerzeroedthick.
8. Protection against space exhaustion—each virtual disk type, based on its architecture, has varying degrees of protection against space exhaustion. Thin virtual disks do not reserve space on the VMFS datastore upon creation and instead grow in 1 MB blocks as needed. Therefore, if unmonitored, as one or more thin virtual disks grow on the datastore, they could exhaust the capacity of the VMFS. Even if the underlying array has plenty of additional capacity to provide. If careful monitoring is in place that provides the ability to make proactive resolution of capacity exhaustion (moving the virtual machines around or grow the VMFS) thin virtual disks are a perfectly acceptable choice. Storage DRS is an excellent solution for space exhaustion prevention. While careful monitoring can protect against this possibility, it can still be of a concern and should be contemplated upon initial provisioning. Zeroedthick and eagerzeroedthick virtual disks are not susceptible to VMFS logical capacity exhaustion because the space is reserved on the VMFS upon creation. Recommendation: eagerzeroedthick.
9. Virtual disk density—it should be noted that while all virtual disk types take up the same amount of physical space on the FlashArray due to data reduction, they have different requirements on the VMFS layer. Thin virtual disks can be oversubscribed (more capacity provisioned than the VMFS reports as being available) allowing for far more virtual disks to fit on a given volume than either of the thick formats. This provides a greater virtual machine to VMFS datastore density and reduces the number or size of volumes that are required to store them. This, in effect, reduces the management overhead of provisioning and managing additional volumes in a VMware environment. Recommendation: thin.
10. Time to create—the virtual disk types also vary in how long it takes to initially create them. Since thin and zeroedthick virtual disks do not zero space until they are actually written to by a guest they are both created in trivial amounts of time—usually a second or two. Eagerzeroedthick disks, on the other hand, are pre-zeroed at creation and consequently take additional time to create. If the time-to-first-IO is paramount for whatever reason, thin or zeroedthick is best. Recommendation: thin.
11. Space efficiency—the aforementioned bullet on “virtual disk density” describes efficiency on the VMFS layer. Efficiency on the underlying array should also be considered. In vSphere 6.0, thin virtual disks support guest-OS initiated UNMAP to a virtual disk, through the VMFS and down to the physical storage. Therefore, thin virtual disks can be more space efficient as time wears on and data is written and deleted. For more information on this functionality in vSphere 6.0, refer to the section, In-Guest UNMAP in ESXi 6.x, that can be found later in this paper. Recommendation: thin.
12. Storage usage trending—A useful metric to know and track is how much capacity is actually being used by a virtual machine guest. If you know how much space is being used by the guests, and furthermore, at what rate that is growing, you can more appropriately size and project storage allocations. Since thick type virtual disks reserve all of the space on the VMFS whether or not the guest has used it, it is difficult to know, without guest tools, how much the guest has actually written. Often it is not known until the application has used its available space and the administrator requests more. This leads to abrupt and unplanned capacity increases. Thin virtual disks only reserve what the guest has written, therefore will grow as the guest adds more data. This growth can be monitored and trended. This will allow VMware administrators to plan and predict future storage needs. Recommendation: thin.
    Note:

    BEST PRACTICE: Use thin virtual disks for most virtual machines. Use eagerzeroedthick for virtual machines that require very high performance levels.

    Do not use zeroedthick.

    No virtual disk option quite fits all possible use-cases perfectly, so choosing an allocation method should generally be decided upon on a case-by-case basis. VMs that are intended for short term use, without extraordinarily high performance requirements, fit nicely with thin virtual disks. For VMs that have higher performance needs eagerzeroedthick is a good choice.