Install ESXi 7 on SD Card or USB

There have been a lot of issues and discussion going on about the installation of ESXi 7 on SD Cards or USB devices. Although it was never recommended, many organizations installed ESXi on SD/USB without any issues until ESXi version 7 was released.

While there have been several issues with some ESXi 7 versions, the release of ESXi 7.0U3c resolves all SD/USB related issues. However, this configuration does have a deprecated status and some additional configuration steps are required. Next to the deprecation status of SD/USB for vSphere 7, customers also face the [End of General Support] for vSphere 6.5/6.7 which is 15 October 2022 which can put them in a challenging situation. For example; do you want to invest in local SSD/NVMe for servers that are about to be replaced within 1 year? Or do you want to invest in buying extended support from VMware? The answer to both questions is probably “no”, but you also do not want to be in a position to run vSphere 6.7 without full support.

There are various documents regarding the ESXi 7 on SD Card/USB. The most noticible are [KB85685] and the [VMware Core Documentation]. Although very useful, it was not 100% clear to me how this would work and look like on a 8GB SD/USB. I decided to do some installs to see the exact behavior. For testing purposes, I will create a Nested / virtual ESXi with the 7.0U3f ESXi .iso

The first challenge was to create a virtual ESXi which has a USB Controller. William Lam wrote a [blogpost] about this, however I took a slightly different approach.


Create virtual ESXi Machine

  • Create a new virtual machine (I have named my virtual ESXi host vesxi7-3.)
  • Select Other, VMware ESX 7.0 as Guest OS
  • Remove the SCSI Controller and disk.
  • Add a USB 3.1 Controller.
  • Mount the ESXi 7.0U3f ISO to the virtual CD/DVD of the virtual machine.
  • … do not forget to enable Hardware virtualization on the CPU settings if you want to run some VMs.

After the VM has been created you have to create an empty .vmdk file and add this to the .vmx configuration.

  • SSH to the physical ESXi host and change to the directory of new virtual ESXi VM home directory.
  • Create an empty vmdk file (8GB) with the command:
vmkfstools -c 8G /vmfs/volumes/nfs02/vesxi7-3/vesxi7-3.vmdk
  • Edit the .vmx file to connect the created .vmdk file to the VM configuration.
vi /vmfs/volumes/nfs02/vesxi7-4/vesxi7-4.vmx
  • Add below lines to .vmx file
usb_xhci:0.present = "TRUE"
usb_xhci:0.deviceType = "disk"
usb_xhci:0.fileName = "vesxi7-3.vmdk"
usb_xhci:0.readOnly = "FALSE"
usb_xhci:0.speed = "16"
usb_xhci:0.parent = "-1"
usb_xhci:0.port = "0"
  • Reload VM configuration with command
vim-cmd vmsvc/reload $(vim-cmd vmsvc/getallvms | grep 'vesxi7-3' | awk '{print $1}')

Install ESXi

I don’t think I have to go through the installation of ESXi, but I’ll share this screenshot from the disk overview during installation:

After the installation is complete you are greeted with 4 warning messages (you may have to press ALT+F11 in the VM console to be able to see this. Return to the DCUI with ALT+F2).

  • Secondary persistent device not found. Please move installation to persistent storage as support for SD-Card/USB only configuration is being deprecated.
  • vSphere 8.0 will no longer support SD-Card/USB only configuration, please refer to release notes
  • No persistent storage available for system logs and data. ESX is operating with limited system storage space, logs and system data will be lost on reboot
  • Logs are stored on non-persistent storage. Consult product documentation to configure a syslog server or a scratch partition

Within vCenter, on the virtual ESXi you can see the USB disk (USB 41001) is attached / in use.


Add ESXi to vCenter

When you add the new virtual ESXi host to vCenter you get similar warning messages as in the console:

  • No datastores have been configure
  • No coredump target has been configured. Host core dumps cannot be saved.
  • System logs on host vesxi7-3.infrajedi.local are stored on non-persistent storage.

The first step I took was to add a NFS share named “nfs-zulu” to the virtual ESXi host. This will automatically result in the creation of the vCLS VM and gets rid of the first warning message.


Configure scratch partition

On the “nfs-zulu” NFS share I have created a directory called scratch. Within this directory I have created additional directories for each ESXi server. So /vmfs/volumes/nfs-zulu/scratch/vesxi7-3.

After the creation of these directories, I edited the Advanced Setting ScratchConfig.ConfiguredScratchLocation to point to this new directory:

Reboot for the settings to take effect. The layout of the scratch directory looks as follows:

You can see the scratch partition holds your log files so they won’t be lost upon reboot.

Additionally, I would recommend to configure the host to send its logs to vRealize Log Insight (vRLI) through the advanced Setting (Syslog.global.logHost) or via the vRLI – vSphere integration.


Configure coredump file to USB

After configuring the scratch partition, the only warning message that is left is about the coredump file. For coredump a file or partition can be configured on VMFS. Since I do not have a VMFS datastore, I configured the allowCoreDumponUsb Kernel option. This is a new kernel option as documented in VMware [KB77009] – New Kernel options available on ESXi 7.0

  • Edit the Advanced System Settings on your host and change the value of key VMkernel.Boot.allowCoreDumpOnUsb to true
  • Reboot your server

All error messages in vCenter should be gone now.

I you want to see if/how the coredump configuration works, you can force a panic with the command:

vsish -e set /reliability/crashMe/Panic 1

If you wonder about VMware Tools on SD/USB; as of ESXi 7.0U3c, VMware Tools are automatically loaded to RAMdisk. You may want to have a look at [this] blogpost to setup a central VMware Tools repository.


Command line output to give some more insight:

[root@vesxi7-3:/vmfs/volumes] df -h
Filesystem   Size   Used Available Use% Mounted on
VMFS-L       6.8G   1.6G      5.2G  24% /vmfs/volumes/LOCKER-62e03282-31ce2dbe-23e8-00505686cf0c
vfat       499.7M 201.4M    298.3M  40% /vmfs/volumes/BOOTBANK1
vfat       499.7M   8.0K    499.7M   0% /vmfs/volumes/BOOTBANK2

List disk devices Names:
[root@vesxi7-3:~] ls -ltrh /vmfs/devices/disks/
total 16773104
lrwxrwxrwx    1 root     root          22 Jul 26 19:07 vml.0000000000766d68626133323a303a30:7 -> mpx.vmhba32:C0:T0:L0:7
lrwxrwxrwx    1 root     root          22 Jul 26 19:07 vml.0000000000766d68626133323a303a30:6 -> mpx.vmhba32:C0:T0:L0:6
lrwxrwxrwx    1 root     root          22 Jul 26 19:07 vml.0000000000766d68626133323a303a30:5 -> mpx.vmhba32:C0:T0:L0:5
lrwxrwxrwx    1 root     root          22 Jul 26 19:07 vml.0000000000766d68626133323a303a30:1 -> mpx.vmhba32:C0:T0:L0:1
lrwxrwxrwx    1 root     root          20 Jul 26 19:07 vml.0000000000766d68626133323a303a30 -> mpx.vmhba32:C0:T0:L0
-rw-------    1 root     root        6.9G Jul 26 19:07 mpx.vmhba32:C0:T0:L0:7
-rw-------    1 root     root      500.0M Jul 26 19:07 mpx.vmhba32:C0:T0:L0:6
-rw-------    1 root     root      500.0M Jul 26 19:07 mpx.vmhba32:C0:T0:L0:5
-rw-------    1 root     root      100.0M Jul 26 19:07 mpx.vmhba32:C0:T0:L0:1
-rw-------    1 root     root        8.0G Jul 26 19:07 mpx.vmhba32:C0:T0:L0

[root@vesxi7-3:~] ls -lha /vmfs/volumes
total 1540
drwxr-xr-x    1 root     root         512 Jul 26 19:04 .
drwxr-xr-x    1 root     root         512 Jul 26 18:33 ..
drwxr-xr-x    1 root     root           8 Jan  1  1970 0129c9b4-8f4e8205-c594-523c32ff4483
drwxr-xr-t    1 root     root       72.0K Jul 26 18:29 62e03282-31ce2dbe-23e8-00505686cf0c
lrwxr-xr-x    1 root     root          35 Jul 26 19:04 BOOTBANK1 -> b2dc09aa-dd4a84c1-45a6-fd1a15cbc410
lrwxr-xr-x    1 root     root          35 Jul 26 19:04 BOOTBANK2 -> 0129c9b4-8f4e8205-c594-523c32ff4483
lrwxr-xr-x    1 root     root          35 Jul 26 19:04 LOCKER-62e03282-31ce2dbe-23e8-00505686cf0c -> 62e03282-31ce2dbe-23e8-00505686cf0c
drwxr-xr-x    1 root     root           8 Jan  1  1970 b2dc09aa-dd4a84c1-45a6-fd1a15cbc410
Overview of RAMdisks with vdf -h command)
Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        3M       28M  12% --
etc                        28M      956K       27M   3% --
opt                        32M        0B       32M   0% --
var                        48M      664K       47M   1% --
tmp                       256M        4K      255M   0% --
iofilters                  32M        0B       32M   0% --
shm                      1024M        0B     1024M   0% --
crx                      1024M        0B     1024M   0% --
configstore                32M      160K       31M   0% --
configstorebkp             32M      160K       31M   0% --
osdata                   2048M        3M     2044M   0% --
tools                     286M      183M      102M  64% --
hostdstats                114M        1M      112M   1% --
files in /var/log are in scratch partition

[root@vesxi7-3:/var/log] ls -lha /var/log
total 640
drwxr-xr-x    1 root     root         512 Jul 26 18:53 .
drwxr-xr-x    1 root     root         512 Jul 26 18:33 ..
-rw-r--r--    1 root     root         523 Jul 26 18:35 .vmsyslogd.err
lrwxrwxrwx    1 root     root          23 Jul 26 18:33 LogEFI.log -> /scratch/log/LogEFI.log
lrwxrwxrwx    1 root     root          21 Jul 26 18:33 Xorg.log -> /scratch/log/Xorg.log
lrwxrwxrwx    1 root     root          29 Jul 26 18:33 apiForwarder.log -> /scratch/log/apiForwarder.log
lrwxrwxrwx    1 root     root          24 Jul 26 18:33 attestd.log -> /scratch/log/attestd.log
lrwxrwxrwx    1 root     root          21 Jul 26 18:33 auth.log -> /scratch/log/auth.log
-rw-rw-rw-    1 root     root       52.8K Jul 26 18:33 boot.gz
lrwxrwxrwx    1 root     root          22 Jul 26 18:33 clomd.log -> /scratch/log/clomd.log
<…cutted>

Henk Engelsman

Leave a Reply

Your email address will not be published.

Post comment