ESXi Config Backup & Restore

The state.tgz file

One nice thing only a very few people know is the configuration file “state.tgz” of ESXi. If you’re using a SD-Card or USB Stick installation of ESXi, the boot image is stored on that USB / SD Card.

The boot process

During bootup, ESXi loads the configuration from /bootbank/state.tgz and extracts its contents to /etc. During the day, if the file gets updated, changes are backed up into state.tgz and placed to /bootbank again.

Advantage

Imagine the day, ESXi tells you there’s something wrong with /bootbank. At a closer look you see the SD Card suddenly died today. First – yes realize that – you’re happy ESXi and all VM’s are still up and running. You’ll be able to vMotion everything off that host and put him into maintenance mode.

Now, this is the right and latest time to create a config backup – if you didn’t before πŸ™‚

Create a config Backup

To backup your ESXi configuration from USB / SD Card, follow this steps:

  1. Logon to your ESXi Host using SSH
  2. Run the auto-backup.sh script to confirm you have an up-to-date host configuration saved in the /bootbank/state.tgz file
  3. scp the file /bootbank/state.tgz to some safe location

Reinstall ESXi

If you’re in the same situation like me, it’s now time to shutdown the Host and reboot. You’ll see, the Host will not come up – yes because of the defective SD. πŸ˜› Now insert a new SD Card, use ESXi boot media / installer and start from scratch.

After ESXi is installed again, just give him an IP and root passwort to be able to connect using ssh and continue with the restore process.

Restore Configuration

To restore the configuration:

  • scp the backup state.tgz to /tmp
  • Login using ssh
  • cd /tmp
  • tar -xvzf /tmp/state.tgz
  • cd /
  • mv local.tgz local.tgz.old
  • cp /tmp/local.tgz .
  • tar -xzvf local.tgz
  • Reboot the ESXi Host

The Host now starts with the restored configuration from your state.tgz file.

Source:
http://kb.vmware.com/kb/2043048

Advertisements

ESXi Pink Screen of Death (PsOD)

Have you ever seen one? πŸ™‚

This was my first:

psod

How it started

But let me start from the beginning. We’re using Veeam Backup Software installed as a VM, backing up all other VM’s on the same and another host in a shared vCenter environment. As soon as Veeam tried to backup VM’s from the same host, ESXi ran into this pink screen.

Analysis

The Screenshot tells us something about Exception 14 in world 34461 (line 2) and even more informational, there’s another hint “…E1000PollRxRing@vmkernel…”. You’ll find this at the first line beginning with 0x41239a…

Solution

VMware recommends using vmxnet3 Network Adapters every possible time. We didn’t and actually ran into a problem here. Since we’ve changed only the network card of the Veeam VM to vmxnet3, we never saw Pink screens again. Unfortunately. πŸ˜›

Upgrade ESXi to 5.5 using a USB Stick

USB Stick vorbereiten

  • Open cmd
  • Start diskpart
  • list disk (to show up all existing disks)
  • select disk 1 (where number 1 is the number of your USB Stick
  • clean (to clear the existing config)
  • create partition primary
  • select partition 1
  • active
  • format fs=fat32 quick
  • assign
  • exit

After this steps:

Interactive Upgrade

  • Boot from USB Installation Media

If the installer finds an existing ESX or ESXi installation and VMFS datastore you can choose from the following options:

  • Upgrade ESXi, preserve VMFS datastore
  • Install ESXi, preserve VMFS datastore
  • Install ESXi, overwrite VMFS datastore

INFO: If an existing VMFS datastore cannot be preserved, you can choose only to install ESXi and overwrite the existing VMFS datastore, or to cancel the installation. If you choose to overwrite the existing VMFS datastore, back up the datastore first.

  • Press F11 to confirm and start the upgrade
  • At the end, Remove USB Stick, Press Enter to reboot the host

After the Upgrade

  • Connect to Host
  • View the upgrade logs. You can use the vSphere Web Client to export the log files.
  • Import Licenses and reapply them to ESX Hosts. When the upgrade is complete, ESXi is in evaluation mode.

vSphere Web Client vs. vSphere Client Appl. (c#)

VMware itself tells us to migrate to the new Web Client, because many of the new features are only available on the Webinterface and not integrated into the old vSphere Client Application.

BUT: there are also features, you’ll miss in the Web Client. You’d like to connect a ESX directly? Not possible. You like the Maps feature? You’ll miss it. Shame on you VMware.

Here’s my list of Features.

Only in vSphere Client Application

  • Maps for vCenter
  • old Plugins (most will work, not not all)
  • direct connect to ESX Host
  • Custom Attributes
  • Update Manager, editing Baselines

Only in Web Client

  • Tags
  • vSphere Replication
  • shared nothing vMotion
  • vmdk >2TB
  • vm Hardware v10

References

Florian Grehl created a more detailed list at his blog virten.net:
http://www.virten.net/2013/04/vsphere-web-client-vs-vsphere-client-c-client/

VMware itself also published a document “Which vSphere client should I use?”:
http://blogs.vmware.com/vsphere/files/2012/11/vSphere-client-comparison.pdf

About Update Manager:
http://pubs.vmware.com/vsphere-55/topic/com.vmware.vsphere.update_manager.doc/GUID-255001C1-B080-4789-B157-3523E7B80E1A.html

VMDirectPath owns the local SCSI Controller

Yesterday I was witness of an exciting feature of VMware’s VMDirectPath Feature on ESXi. For those who don’t know: DirectPath allows you to directly attach PCI Devices to a VM.

In our case we installed a secondary SCSI Adapter to attach a tape drive to a VM. In ESX Host that is configured by using VMDirectPath. Unfortunately, fast hands have choosen the wrong adapter. In this case, where you only have two adapters, it’s the SCSI adapter of the disks where ESX ist installed on πŸ˜› it isn’t possible to write down any configuration changes to disk now, because esx has no longer access to the SCSI Controller.

ESX boots up normally. After OS is up, the SCSI Adapter is passed trough to the DirectPath module. From this time on, you see error messages in the logs (ALT+F12 on console).

So, how to get back?

  • Boot another Linux Live CD and change the configuration? Didn’t work in our case, why ever.
  • Reinstall ESX: no.
  • Change the configuration during ESX operation. But how if filesystem is read only?

Here’s my How-To

Run the following on ESX Console (dcui) to get a list of who owns which hba:

pic1

Now let’s assign vmhba1 back to vmkernel:

pic2

Now kernel owns the hba, but before changes can be written to disk, a rescan is required.

pic3

Now let’s modify the esx.conf to permanentely assign vmhba1 to vmkernel.

pic4

To search a String, use “/” in vi; just type “/vmhba1” and hit enter, vi will show up the right line where you can see vmhba1 is assigned to passtrough instead of vmkernel. Some other vi hints for editing:

  • delete text using the DEL key
  • before typing, enter insert mode by pressing key “i”
  • after typing exit insert mode using ESC key
  • to save and exit: “:wq” and ENTER

Now do a reboot to test the new setting. After all now we’re able to assign the next/right hba to the passtrough πŸ˜‰

pic5

VMware Workstation with Linux Mint Kernel 3.0.0

Since Christmas 2011 I’m trining to work on Linux Mint at home. It was clear that it isn’t possible living without any Windows Operation system installed, so I’ve decided to use Windows 7 in a virtual Machine. In VMware Workstation there’s a very useful feature called “Unity”, that makes all Windows Applications seamless available in the host operation system. That’s what I need to have the best feeling in the daily business.

After installing VMware Workstation, the first start already stopped with a error message, that I had to search in /tmp/vmware-root/setup-xxxx.log. At the end of the Logfile was a message like “Failed to compile module vmmon!”.

Google told me this is a known error under different Kernel Versions. In my case, I’m using this version:

# uname -a
Linux laptop7 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

In VMware’s communities was a post explaining how to Patch my problem:
http://communities.vmware.com/thread/326180

Here’s how to in quick steps:

After this steps, the VMware modules can be compiled into the Kernel using the following command or by starting VMware Workstation again.

  • vmware-modconfig –console –install-all