vSphere Memory Tiering with Asus Hyper M.2 X16 for nested VCF 9

As written in the previous blog, I was quite satisfied with my homelab setup. It doesn’t take to much space, is low noise and has enough resources and performance to run a Nested VCF 5 setup. With the release of VCF 9, I am still able to run a basic setup, but with the additional VCF Automation and Supervisor components, I required a bit more resources, especially memory.

My first idea was to buy additional memory for the HPE Z8 G4 which is of type DDR4-2666 ECC Reg. Over here the costs would be around €200 for 128GB, totaling 512 GB in my system.

Then my idea shifted towards using Memory Tiering. This functionality is available as tech preview in vSphere 8.0 U3. See the announcement [blog]. Since I didn’t have any free storage connections available in my system, my eye fell on the the [Asus Hyper M.2 X16]. This is a PCIE X16 card that can fit 4 M.2 SSDs. I had some doubts about this solution, since vSphere Memory Tiering on ESXi 8 is in Tech Preview and I read some confusing articles on bifurcation as well as some discussion whether or not nested ESX 9 would work. Long story shorts… I ended up buying the Asus Card along with a 2 TB Samsung 990 Pro M.2. In case this wouldn’t work as hoped, I could always use the Samsung as regular storage device. Costs for this solution was around the some amount as extra RAM, with the Asus card around €50 and the SSD around €160.

Asus Hyper M.2 X16 and Samsung 990 Pro Installation

The card is neatly packed and comes in a nice box as expected from Asus. The card comes with 4 small plastic bags containing screw sets for mounting the M.2 SSD. The installation steps I followed were:

• Remove the heatsink (4 black screws)
• Install the bottom screw on the top of the card (where the end of the SSD will rest)

• Install the M.2 SSD
• Fix the M.2 SDD with the second crew
• Remove the protection sticker from the correct heat mat

• Remount the heatsink
• Turn on the Fan switch
• Install the card in the system

Easy as that 🙂

Configure Memory Tiering on ESXi 8

My initial setup has 384 GB of RAM as you can see in the screenshot below.

The first step before enabling memory tiering is to determine to location of the M.2 SSD that is installed on the Asus card. You can find it by checking the host’s storage devices and look for the new unused M.2 SSD.

Copy the Location of the M.2 SSD and open an SSH session to your host.

Enable Memory tiering on the host with the command:

esxcli system settings kernel set -s MemoryTiering -v TRUE

No output is returned

Configure the device to be used for memory tiering with the command:

esxli system tierdevice create -d <location of SSD device>

…where <location of SSD device> is the value we captured earlier, so

/vmfs/devices/disks/t10.NVMe____NVMe____Samsung_SSD_990_PRO_2TB__________29F2405144382500

With memory tiering enabled, we now need to configure the percentage of installed physical RAM to be added from the M.2 SSD, with 400 being the maximum value.

Since I have 384 GB RAM, I decided to set the value to 200%, adding 768GB of RAM, totaling 1024GB. This means that more of the half of the SSD’s capacity will not be used (for now). See below screenshots of the commands executed.

Last action is to reboot the system for the settings to go in effect.

As you can see in the screenshot below Memory now shows 1,12 TB capacity

Testing Nested VCF 9 with Memory Tiering

To do some basic testing, I created 4 ESX 9 VMs, each with 8 vCPU and 96GB of RAM. ESX installation was done by go-via (see my previous [blog]). Since this is not the latest and greatest hardware (CPU), I had to configure all settings below in my ks.cfg to be able to install:

--ignoreprereqwarnings
--ignoreprereqerrors
--forceunsupportedinstall

Otherwise no issues were seen during installation.

After ESX installation, I deployed the VCF Installer appliance (on the physical host) to deploy a VCF 9 Fleet. For now just a small fleet, with the basic components;

• vCenter
• NSX
• SDDC Manager
• VCF Operations (+ Collector)
• VCF Fleet Manager

Although I did not do any extensive testing, I did not run into any issues during the deployment of VCF 9.

Conclusion

• The Asus HyperX card works in my system, but it’s usage may be limited on other systems (bifurcation)
• The Asus HyperX card with Samsung 990 Pro can be used for memory tiering.
• Memory tiering can be used to host nested ESX 9.

Next steps

Next planned steps include deploying more of the VCF 9 solution and thereby generating extra load on the system:

• Deploy VCF Automation
• Deploy Workload domain
• Enable Supervisor

References

[KB311934] Using the “Memory Tiering over NVMe” feature in vSphere 8.0 Update 3. At the bottom of the KB article there is a nice .pdf file “Memory Tiering over NVMe Tech Preview – vSphere 8.0.3 Technical Guide Revision 1.2.pdf” with additional information.