Install ESXi 7 on SD Card or USB
There have been a lot of issues and discussion going on about the installation of ESXi 7 on SD Cards or USB devices. Although it was never recommended, many organizations installed ESXi on SD/USB without any issues until ESXi version 7 was released.
While there have been several issues with some ESXi 7 versions, the release of ESXi 7.0U3c resolves all SD/USB related issues. However, this configuration does have a deprecated status and some additional configuration steps are required. Next to the deprecation status of SD/USB for vSphere 7, customers also face the [End of General Support] for vSphere 6.5/6.7 which is 15 October 2022 which can put them in a challenging situation. For example; do you want to invest in local SSD/NVMe for servers that are about to be replaced within 1 year? Or do you want to invest in buying extended support from VMware? The answer to both questions is probably “no”, but you also do not want to be in a position to run vSphere 6.7 without full support.
There are various documents regarding the ESXi 7 on SD Card/USB. The most noticible are [KB85685] – SD card/USB boot device revised guidance and the [VMware Core Documentation]. Although very useful, it was not 100% clear to me how this would work and look like on a 8GB SD/USB. I decided to do some installs to see the exact behavior. For testing purposes, I will create a Nested / virtual ESXi with the 7.0U3f ESXi .iso
The first challenge was to create a virtual ESXi which has a USB Controller. William Lam wrote a [blogpost] about this, however I took a slightly different approach.
Create virtual ESXi Machine
- Create a new virtual machine (I have named my virtual ESXi host vesxi7-3.)
- Select Other, VMware ESX 7.0 as Guest OS
- Remove the SCSI Controller and disk.
- Add a USB 3.1 Controller.
- Mount the ESXi 7.0U3f ISO to the virtual CD/DVD of the virtual machine.
- … do not forget to enable Hardware virtualization on the CPU settings if you want to run some VMs.
After the VM has been created you have to create an empty .vmdk file and add this to the .vmx configuration.
- SSH to the physical ESXi host and change to the directory of new virtual ESXi VM home directory.
- Create an empty vmdk file (8GB) with the command:
vmkfstools -c 8G /vmfs/volumes/nfs02/vesxi7-3/vesxi7-3.vmdk- Edit the .vmx file to connect the created .vmdk file to the VM configuration.
vi /vmfs/volumes/nfs02/vesxi7-4/vesxi7-4.vmx- Add below lines to .vmx file
usb_xhci:0.present = "TRUE"
usb_xhci:0.deviceType = "disk"
usb_xhci:0.fileName = "vesxi7-3.vmdk"
usb_xhci:0.readOnly = "FALSE"
usb_xhci:0.speed = "16"
usb_xhci:0.parent = "-1"
usb_xhci:0.port = "0"
- Reload VM configuration with command
vim-cmd vmsvc/reload $(vim-cmd vmsvc/getallvms | grep 'vesxi7-3' | awk '{print $1}')Install ESXi
I don’t think I have to go through the installation of ESXi, but I’ll share this screenshot from the disk overview during installation:
After the installation is complete you are greeted with 4 warning messages (you may have to press ALT+F11 in the VM console to be able to see this. Return to the DCUI with ALT+F2).
Secondary persistent device not found. Please move installation to persistent storage as support for SD-Card/USB only configuration is being deprecated.vSphere 8.0 will no longer support SD-Card/USB only configuration, please refer to release notesNo persistent storage available for system logs and data. ESX is operating with limited system storage space, logs and system data will be lost on rebootLogs are stored on non-persistent storage. Consult product documentation to configure a syslog server or a scratch partition
Within vCenter, on the virtual ESXi you can see the USB disk (USB 41001) is attached / in use.
Add ESXi to vCenter
When you add the new virtual ESXi host to vCenter you get similar warning messages as in the console:
- No datastores have been configure
- No coredump target has been configured. Host core dumps cannot be saved.
- System logs on host vesxi7-3.infrajedi.local are stored on non-persistent storage.
The first step I took was to add a NFS share named “nfs-zulu” to the virtual ESXi host. This will automatically result in the creation of the vCLS VM and gets rid of the first warning message.
Configure scratch partition
On the “nfs-zulu” NFS share I have created a directory called scratch. Within this directory I have created additional directories for each ESXi server. So /vmfs/volumes/nfs-zulu/scratch/vesxi7-3.
After the creation of these directories, I edited the Advanced Setting ScratchConfig.ConfiguredScratchLocation to point to this new directory:
Reboot for the settings to take effect. The layout of the scratch directory looks as follows:
You can see the scratch partition holds your log files so they won’t be lost upon reboot.
Additionally, I would recommend to configure the host to send its logs to vRealize Log Insight (vRLI) through the advanced Setting (Syslog.global.logHost) or via the vRLI – vSphere integration.
Configure coredump file to USB
After configuring the scratch partition, the only warning message that is left is about the coredump file. For coredump a file or partition can be configured on VMFS. Since I do not have a VMFS datastore, I configured the allowCoreDumponUsb Kernel option. This is a new kernel option as documented in VMware [KB77009] – New Kernel options available on ESXi 7.0
- Edit the Advanced System Settings on your host and change the value of key VMkernel.Boot.allowCoreDumpOnUsb to true
- Reboot your server
All error messages in vCenter should be gone now.
I you want to see if/how the coredump configuration works, you can force a panic with the command:
vsish -e set /reliability/crashMe/Panic 1
If you wonder about VMware Tools on SD/USB; as of ESXi 7.0U3c, VMware Tools are automatically loaded to RAMdisk. You may want to have a look at [this] blogpost to setup a central VMware Tools repository.
Command line output to give some more insight:
[root@vesxi7-3:/vmfs/volumes] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-L 6.8G 1.6G 5.2G 24% /vmfs/volumes/LOCKER-62e03282-31ce2dbe-23e8-00505686cf0c
vfat 499.7M 201.4M 298.3M 40% /vmfs/volumes/BOOTBANK1
vfat 499.7M 8.0K 499.7M 0% /vmfs/volumes/BOOTBANK2
List disk devices Names:
[root@vesxi7-3:~] ls -ltrh /vmfs/devices/disks/
total 16773104
lrwxrwxrwx 1 root root 22 Jul 26 19:07 vml.0000000000766d68626133323a303a30:7 -> mpx.vmhba32:C0:T0:L0:7
lrwxrwxrwx 1 root root 22 Jul 26 19:07 vml.0000000000766d68626133323a303a30:6 -> mpx.vmhba32:C0:T0:L0:6
lrwxrwxrwx 1 root root 22 Jul 26 19:07 vml.0000000000766d68626133323a303a30:5 -> mpx.vmhba32:C0:T0:L0:5
lrwxrwxrwx 1 root root 22 Jul 26 19:07 vml.0000000000766d68626133323a303a30:1 -> mpx.vmhba32:C0:T0:L0:1
lrwxrwxrwx 1 root root 20 Jul 26 19:07 vml.0000000000766d68626133323a303a30 -> mpx.vmhba32:C0:T0:L0
-rw------- 1 root root 6.9G Jul 26 19:07 mpx.vmhba32:C0:T0:L0:7
-rw------- 1 root root 500.0M Jul 26 19:07 mpx.vmhba32:C0:T0:L0:6
-rw------- 1 root root 500.0M Jul 26 19:07 mpx.vmhba32:C0:T0:L0:5
-rw------- 1 root root 100.0M Jul 26 19:07 mpx.vmhba32:C0:T0:L0:1
-rw------- 1 root root 8.0G Jul 26 19:07 mpx.vmhba32:C0:T0:L0
[root@vesxi7-3:~] ls -lha /vmfs/volumes
total 1540
drwxr-xr-x 1 root root 512 Jul 26 19:04 .
drwxr-xr-x 1 root root 512 Jul 26 18:33 ..
drwxr-xr-x 1 root root 8 Jan 1 1970 0129c9b4-8f4e8205-c594-523c32ff4483
drwxr-xr-t 1 root root 72.0K Jul 26 18:29 62e03282-31ce2dbe-23e8-00505686cf0c
lrwxr-xr-x 1 root root 35 Jul 26 19:04 BOOTBANK1 -> b2dc09aa-dd4a84c1-45a6-fd1a15cbc410
lrwxr-xr-x 1 root root 35 Jul 26 19:04 BOOTBANK2 -> 0129c9b4-8f4e8205-c594-523c32ff4483
lrwxr-xr-x 1 root root 35 Jul 26 19:04 LOCKER-62e03282-31ce2dbe-23e8-00505686cf0c -> 62e03282-31ce2dbe-23e8-00505686cf0c
drwxr-xr-x 1 root root 8 Jan 1 1970 b2dc09aa-dd4a84c1-45a6-fd1a15cbc410
Overview of RAMdisks with vdf -h command)
Ramdisk Size Used Available Use% Mounted on
root 32M 3M 28M 12% --
etc 28M 956K 27M 3% --
opt 32M 0B 32M 0% --
var 48M 664K 47M 1% --
tmp 256M 4K 255M 0% --
iofilters 32M 0B 32M 0% --
shm 1024M 0B 1024M 0% --
crx 1024M 0B 1024M 0% --
configstore 32M 160K 31M 0% --
configstorebkp 32M 160K 31M 0% --
osdata 2048M 3M 2044M 0% --
tools 286M 183M 102M 64% --
hostdstats 114M 1M 112M 1% --
files in /var/log are in scratch partition
[root@vesxi7-3:/var/log] ls -lha /var/log
total 640
drwxr-xr-x 1 root root 512 Jul 26 18:53 .
drwxr-xr-x 1 root root 512 Jul 26 18:33 ..
-rw-r--r-- 1 root root 523 Jul 26 18:35 .vmsyslogd.err
lrwxrwxrwx 1 root root 23 Jul 26 18:33 LogEFI.log -> /scratch/log/LogEFI.log
lrwxrwxrwx 1 root root 21 Jul 26 18:33 Xorg.log -> /scratch/log/Xorg.log
lrwxrwxrwx 1 root root 29 Jul 26 18:33 apiForwarder.log -> /scratch/log/apiForwarder.log
lrwxrwxrwx 1 root root 24 Jul 26 18:33 attestd.log -> /scratch/log/attestd.log
lrwxrwxrwx 1 root root 21 Jul 26 18:33 auth.log -> /scratch/log/auth.log
-rw-rw-rw- 1 root root 52.8K Jul 26 18:33 boot.gz
lrwxrwxrwx 1 root root 22 Jul 26 18:33 clomd.log -> /scratch/log/clomd.log
<…cutted>
