The state.tgz file
One nice thing only a very few people know is the configuration file “state.tgz” of ESXi. If you’re using a SD-Card or USB Stick installation of ESXi, the boot image is stored on that USB / SD Card.
The boot process
During bootup, ESXi loads the configuration from /bootbank/state.tgz and extracts its contents to /etc. During the day, if the file gets updated, changes are backed up into state.tgz and placed to /bootbank again.
Imagine the day, ESXi tells you there’s something wrong with /bootbank. At a closer look you see the SD Card suddenly died today. First – yes realize that – you’re happy ESXi and all VM’s are still up and running. You’ll be able to vMotion everything off that host and put him into maintenance mode.
Now, this is the right and latest time to create a config backup – if you didn’t before 🙂
Create a config Backup
To backup your ESXi configuration from USB / SD Card, follow this steps:
- Logon to your ESXi Host using SSH
- Run the auto-backup.sh script to confirm you have an up-to-date host configuration saved in the /bootbank/state.tgz file
- scp the file /bootbank/state.tgz to some safe location
If you’re in the same situation like me, it’s now time to shutdown the Host and reboot. You’ll see, the Host will not come up – yes because of the defective SD. 😛 Now insert a new SD Card, use ESXi boot media / installer and start from scratch.
After ESXi is installed again, just give him an IP and root passwort to be able to connect using ssh and continue with the restore process.
To restore the configuration:
- scp the backup state.tgz to /tmp
- Login using ssh
- cd /tmp
- tar -xvzf /tmp/state.tgz
- cd /
- mv local.tgz local.tgz.old
- cp /tmp/local.tgz .
- tar -xzvf local.tgz
- Reboot the ESXi Host
The Host now starts with the restored configuration from your state.tgz file.
Have you ever seen one? 🙂
This was my first:
How it started
But let me start from the beginning. We’re using Veeam Backup Software installed as a VM, backing up all other VM’s on the same and another host in a shared vCenter environment. As soon as Veeam tried to backup VM’s from the same host, ESXi ran into this pink screen.
The Screenshot tells us something about Exception 14 in world 34461 (line 2) and even more informational, there’s another hint “…E1000PollRxRing@vmkernel…”. You’ll find this at the first line beginning with 0x41239a…
VMware recommends using vmxnet3 Network Adapters every possible time. We didn’t and actually ran into a problem here. Since we’ve changed only the network card of the Veeam VM to vmxnet3, we never saw Pink screens again. Unfortunately. 😛
USB Stick vorbereiten
- Open cmd
- Start diskpart
- list disk (to show up all existing disks)
- select disk 1 (where number 1 is the number of your USB Stick
- clean (to clear the existing config)
- create partition primary
- select partition 1
- format fs=fat32 quick
After this steps:
- Boot from USB Installation Media
If the installer finds an existing ESX or ESXi installation and VMFS datastore you can choose from the following options:
- Upgrade ESXi, preserve VMFS datastore
- Install ESXi, preserve VMFS datastore
- Install ESXi, overwrite VMFS datastore
INFO: If an existing VMFS datastore cannot be preserved, you can choose only to install ESXi and overwrite the existing VMFS datastore, or to cancel the installation. If you choose to overwrite the existing VMFS datastore, back up the datastore first.
- Press F11 to confirm and start the upgrade
- At the end, Remove USB Stick, Press Enter to reboot the host
After the Upgrade
- Connect to Host
- View the upgrade logs. You can use the vSphere Web Client to export the log files.
- Import Licenses and reapply them to ESX Hosts. When the upgrade is complete, ESXi is in evaluation mode.
Yesterday I was witness of an exciting feature of VMware’s VMDirectPath Feature on ESXi. For those who don’t know: DirectPath allows you to directly attach PCI Devices to a VM.
In our case we installed a secondary SCSI Adapter to attach a tape drive to a VM. In ESX Host that is configured by using VMDirectPath. Unfortunately, fast hands have choosen the wrong adapter. In this case, where you only have two adapters, it’s the SCSI adapter of the disks where ESX ist installed on 😛 it isn’t possible to write down any configuration changes to disk now, because esx has no longer access to the SCSI Controller.
ESX boots up normally. After OS is up, the SCSI Adapter is passed trough to the DirectPath module. From this time on, you see error messages in the logs (ALT+F12 on console).
So, how to get back?
- Boot another Linux Live CD and change the configuration? Didn’t work in our case, why ever.
- Reinstall ESX: no.
- Change the configuration during ESX operation. But how if filesystem is read only?
Here’s my How-To
Run the following on ESX Console (dcui) to get a list of who owns which hba:
Now let’s assign vmhba1 back to vmkernel:
Now kernel owns the hba, but before changes can be written to disk, a rescan is required.
Now let’s modify the esx.conf to permanentely assign vmhba1 to vmkernel.
To search a String, use “/” in vi; just type “/vmhba1” and hit enter, vi will show up the right line where you can see vmhba1 is assigned to passtrough instead of vmkernel. Some other vi hints for editing:
- delete text using the DEL key
- before typing, enter insert mode by pressing key “i”
- after typing exit insert mode using ESC key
- to save and exit: “:wq” and ENTER
Now do a reboot to test the new setting. After all now we’re able to assign the next/right hba to the passtrough 😉