For a number of years VMware has been using an internal tool to help customers diagnose problems with virtual machines. The tool, named vmss2core, became publically available with the release of VMware Workstation 7.0.
The biggest difference between vmss2core and other debugging solutions is that it requires absolutely no modifications in the virtual machine. No Windows registry keys to change, no Linux kernel modules to configure and no additional software to be installed.
This turned out to be very helpful, as most of production VMs run with no debugging tools enabled. When Windows throws a BSOD, or Linux kernel panics, or either of them hang, there is very little information that can be obtained. True, both Windows and Linux provide extensive debugging tools, but they are frequently cumbersome to setup, consume VM resources and/or pose security threats.
The vmss2core avoids these drawbacks by building on the checkpoint infrastructure in VMware products. If you create a checkpoint of VM or suspend it when problem occurs, the resulting checkpoint files have all the information that debuggers need. All we needed was a utility that converts VMware formats into something that 3rd party debuggers understand. This is precisely what vmss2core does.
Although vmss2core ships with Workstation 7, it is capable of analyzing checkpoints and suspended VMs created by many of VMware earlier products. It works with both with desktop products , e.g. Workstation, and ESX. All you need to do is to create a checkpoint or suspend a VM, then copy out files corresponding to checkpoint to Linux and Windows Host, and run vmss2core with option specifying what output you want to get.
Presently, vmss2core can produce windbg-compatible .dmp file for Windows VMs (-Wxxx), RedHat crash-compatible core files (-Nxxx), MacOS VM formats (-Xxxx), as well as “flat address space view” (no options) and “physical memory view” (-M) formats suitable for analysis with gdb. The utility prints out details for these options when you run it without arguments.
The utility has to guess kernel build numbers and location of debugging-related data structures. Different options control the guessing algorithms. For crash-compatible output, you can either specify linux family using -N4 or -N6 options, or use combination of -N and -l. The argument for -l is generated using the same script as used by built-in debugger. For Windows VM, using "-W" works most of the time. If windbg is not completely happy with the resulting file, you may want to supply more information, e.g. build number with -Wnum, or location of debug block using -WDDB.
The tool uses only part of the checkpoint as an input. For ESX products, that would be .vmss file. For desktop products, it needs a pair of files: .vmss or .vmsn file with virtual device information, and .vmem file with memory image. The vm-script on ESX has an option that will suspend a VM and package .vmss file into the support archive.
What else is this tool good for? Forensics. Since nothing is tweaked in the VM , there is no way to detect from inside VM that analysis is performed. I hope this makes vmss2core an interesting tool for anti-virus/malware/worms community, especially combined with honeypots built using VMware VMs.
The vmss2core may also be of interest to virtual appliance providers. Instead of building remote debugging capabilities, they can request a compressed checkpoint of the VM to be sent to them for analysis.
I am really happy that VMware released this tool, and I hope the customers will find it useful. You can download Workstation 7 with vmss2core here. Please feel free to ask about the tool on VMware forums. Please note that utility may not be officially supported by VMware.
