Backing up the ESXi System Image

Alan Renouf is a community peer for VMware PowerCLI, Virtu-Al.Net.  In 2009, he was named a vEXPERT by VMware.  Alan's main focus is to teach and help other VMware admins automate their VMware infrastructures and make their lives easier through automation.
www.virtu-al.net.

ESXi is based on a system image, VMware used to call this a firmware but some people found this misleading, most people still refer to it as firmware but in this post I will refer to it as a system image.

The system image is a unified image which is the same whether booting from USB, Hard Disk, PXE or any other media. The logic in the first boot will provide auto configuration based on the kind of installation you have.

One thing to remember about ESXi is that it is memory based, so once booted the system image is entirely loaded into memory, ESXi doesn't care if the original media disappears after boot, there is no reliance on the boot device for running after booting.

ESXi is essentially made up from a number of tardisks (VM TAR files) or archives, these tardisks are mounted into the empty root filesystem in order of enumeration, one of these is called the state tardisk or state.tgz.

The state tardisk is made up of any file in /etc which is marked as sticky, VMware uses the sticky bit to flag the files which are to be included as part of this tardisk but only if they reside under /etc, these are files like esx.conf, inetd.con, passwd….  These are the files which will be persisted after boot.

As said earlier the entire system image is loaded into memory on boot so how is the state tardisk persisted ?

Periodically there is a deamon in the background which looks for changes in the config files, when these are found the state tardisk is rebuilt and saved back to the bootdisk, so why are these files not written directly to disk each time ?  The main reason is the USB disk, if these files were written constantly to the USB flash disk then the life expectancy of the disk would not be very long at all, VMware worked with a number of USB flash OEM’s to calculate the best frequency to write these files back to disk.  This is key to remember as the frequency is around once every 10 minutes, obviously if the system crashes and reboots you may lose any configuration which may have been made in the previous 10 minutes.

So knowing this we can effectively back up the configuration of our ESXi hosts (- 10 minute changes worst case) simply by backing up the state tardisk, then if the worst happens we can restore this and have our ESXi host with complete configuration restored in a matter of minute.  Luckily PowerCLI makes this very easy for us, the below script can be run as a scheduled task and will backup the state tardisk as often as the scheduled task is run.

Research has been gathered for this post from TA8245 - ESXi Internals by Oliver Cremel