HowTo - Virtualization for Small Enterprises

Small and Medium sized Enterprises, particularly non-profit Organizations, in their need to minimize costs can look to Open Source and freeware tools to provide quite capable computing resources. The solution we have chosen at Holy Trinity provides for email, shared calendar, file storage and a range of other computing resources at essentially no cost. Virtualization provides for simplified backup and hardware consolidation.

Most of the virtualization solutions that are discussed on various websites and forums apply either to home applications, usually where a user wishes to run Windows on top of Linux, or large enterprises where multiple servers support large numbers of users. The small and medium sized enterprises do not often feature.

The following presents the results of our investigations into Open Source and freeware virtualization tools suitable for server consolidation. This will serve also as a howto for installing and managing the virtualized servers, although rapid changes in the development of Open Source tools make it likely that the conclusions described here will become dated fairly soon. The solution is not necessarily optimal but works well.

Contents

Requirements

A typical non-profit organization can benefit from the following IT services:

• Windows desktop. Users demand it, and often the software to be run requires it. Microsoft offers generous discounts for charities, allowing Windows remote desktop services to be provided to staff and volunteers. When there are more than about five users, a Windows Terminal Server Standard Edition can be considered, as it can reduce overall maintenance and administration load. On entry level server hardware it can support 10-15 simultaneous users.

• An Open Source email server is now quite straightforward to set up using one of the modern Unix variants. It has value when the number of users is more than 5, as POP3 or IMAP services provided by an ISP or Web provider are designed mainly for home users and can be somewhat limiting in a business environment.

• Shared calendaring, room and equipment booking and other types of services can be provided through freely available software, often underpinned by a webserver such as Apache.

• A range of network services can also be provided, such as DHCP, DNS, NTP. While Microsoft Server software also provides these, the licence conditions controlling their use in a network can be very restrictive (if the enterprise chooses to apply them strictly). If Microsoft Server is not used, Samba services for Windows can provide centralized directory services for authentication of Windows desktop PCs.

• Centralized network file storage can also be provided through Samba.

• Virtualization can provide benefits of hardware consolidation and a backup and restore strategy that is quite easy to implement. For disaster recovery, it is a matter of a few minutes to move a backup virtual machine to a second server, even having a different hardware configuration, since the hypervisor presents a common virtualized hardware interface to the virtual machines.

Scenario

The implementation we chose initially back in 2000 consisted of two servers running Windows NT and Linux. The Linux server was setup on an old desktop machine and proved valuable as Windows NT would crash almost weekly. Samba on Linux and a small number of standalone Windows XP machines allowed the organization to continue operations while the Windows server was out of action. After upgrade, Windows 2003 proved to be far more reliable but the two servers continued to be used. Linux provides Courier-MTA as an email server, including ClamAV antivirus and Spamassassin antispam measures, along with openLDAP, Samba, Apache, FTP, Radius, DHCP and DNS services. Backup was achieved by copying files between machines and to external USB hard drives, requiring the machines to be taken off-line during the night and carefully interweaved scheduling of the various backup tasks.

While two servers can provide a limited backup of computing resources in case of a major failure, consolidation can save some power costs and provide a simplified backup and disaster recovery solution.

Hypervisor Comparisons

We investigated a number of the better known hypervisors, suitable for supporting servers, that could be made to run on a Linux host. There was no consideration given to using Windows as a host. A bare minimum Linux host is a naturally high performance solution and is Open Source. The need to support virtualized guest servers for office use means that high performance is needed in terms of disk and network access by the VMs, but graphics performance is of much less interest. The virtualization tools selected for investigation were VMware Server 2, VMware ESXi, Xen and KVM. Others appeared to be oriented more towards home applications, or were too limited in the freeware version, or were just too little known.

The characteristics of interest to us are: performance relative to a bare-metal install, growable file based VMs for backup to an external disk drive, ability to restore VMs quickly in case of major corruption or hardware failure, ability to mount virtual disks, ability to manage virtual storage by expanding and shrinking partitions, existence of paravirtualized Windows and Linux drivers, ability to take snapshots (to allow backup without pausing the VM) and whether CPU virtualization extensions are needed for fully virtualized VMs.

Performance was measured with tools such as IOmeter (Windows), nbench, bonnie and a large compile task (Linux). All these measures gave inconsistent results but did provide an overall view of the relative performance compared to a bare metal installation. Performance is a complex concept in virtualization as it must account for interactions between VMs, and so far it does not appear to be well understood. The relative overheads listed below are roughly indicative only. Other characteristics refer to ease of use.

Notes:

1. ESXi has excellent management tools in the commercial version, but otherwise provides limited access to the hypervisor for backup and restore in the freeware version. We did not investigate ESXi further. Disk management and backup/restore for Xen and KVM rely on VMware tools.

2. Snapshots in Xen and KVM are possible only with the qcow2 file format, but take some hours to complete for large VMs (which makes them doubtful qualifiers for the name). This format does not allow extraction of individual OS files without some work.

3. Windows PV drivers for block devices are lacking for KVM, but should appear very soon.

4. VMware Server 2 will run fully virtualized VMs, 32 bit only, on hardware without CPU extensions.

Our conclusion is that VMware Server 2 is excellent for small installations that do not require high performance, while Xen is preferred for more rigorous scenarios. KVM may well rival Xen but is still somewhat immature. We have successfully used VMware server 2 for some time. As we describe below it is possible to setup VMs using the open VMware vmdk file format so that they work in either Xen or VMware Server 2 (and most likely KVM also) without the need for conversion. This will be invaluable for crisis management as older hardware running VMware Server can be retained for backup support. Conversion of an installed physical operating system to a VM can be achieved with the excellent VMware Converter tool. Conversions between a range of different VM file formats used by VMware, Xen and KVM can be achieved with tools from the QEMU project.

Another highly significant solution is VirtualBox from Sun Microsystems. It is not strictly free but the licensing conditions are quite liberal. Its performance for server use is similar to VMware, whereas for desktop use it provides excellent 3D performance. It does not require CPU virtualization extensions for 32 bit clients, and can be used to run Windows on top of Linux for example to provide hardware independence and simplified backup. It can also work with VMware open file formats.

VMware Server 2

VMware Server 2 is an excellent free (but not Open Source) hypervisor that is recommended in scenarios where performance is not uppermost. Its toolset is comprehensive and easy to use. The hypervisor runs guest operating systems as processes in an underlying Linux or Windows host. It supports 32 bit guests on older hardware without CPU virtualization extensions. The vmdk VM files can be made to be preallocated or growable, but there is no support for physical partitions.

We have traditionally used RedHat Fedora for server operating systems, but VMware Server can be installed easily on most distributions, including openSUSE which also supports the rpm package management system.

Before starting with VMware it is important to know of some limitations:

1. A 64 bit guest will only run on a CPU with virtualization extensions. A 32-bit fully virtualized guest such as Windows can be installed on older machines without these extensions.

2. The guest hardware must be set up and configured correctly before installing the guest software. It is often difficult to make changes to the VM infrastructure after the software is installed. This is particularly true for networking.

3. VMware server 2.0.0 and 2.0.1 install and work without too many complications on openSUSE 11 and Fedora 10. They may work on earlier kernels. They so far fail to compile on Fedora 11. Outside of these combinations there are no guarantees of success.

VMware server 2 relies on a browser based Virtual Infrastructure utility or command-line tools for management. VMware Server 2 provides its own webserver through which all management can be done. A web browser can be opened on http port 8222 or https port 8333 to access the VMWare configuration utility. Allow the certificate exception, which is due to a self-signed certificate. When administering over the Internet the connection can be a bit flaky, causing random logouts,  and some patience is needed. Often connection problems can be resolved by restarting the browser or the host or both.

The VMware server rpm (or tarball) is downloaded from the VMware site after obtaining a free licence.

openSUSE 11

OpenSUSE has the advantage of having full Xen support, and can be setup to dual boot to either Xen or VMware Server. You should aim for a minimal installation. On the screen asking for desktop selection, choose the item “other”. It is useful to have some sort of GUI installed, either the lightweight XFCE or a minimal X window system if you can cope with the archaic interface. This will facilitate the use of GUI tools at a console and through SSH. When the installation summary page appears, choose to install virtualization under the “Software” heading. Once installed, the machine should reboot. Login as root.

Call up a terminal and enter the command yast2. Navigate to the Software page, choose "Online Update" and update all packages. The installation of virtualization should have automatically added a network bridge. On the "Network Devices", "Network Settings" page, under Overview, there will appear entries for the bridge (br0) and the ethernet interfaces. Leave the ethernet interfaces as “not configured”. Edit the bridge entry and change to a static IP address as appropriate for a server (this will be needed for remote management access). Set the desired ethernet interfaces to be part of the bridge network. Under "DNS and Routing" add the default gateway and under "Hostname/DNS" set the host name and DNS settings.

SSH should now be accessible over the internal network.

In /boot/grub/menu.lst, add dom0_mem=256M after the kernel /boot/xen.gz line. This will run the host with minimal RAM usage, leaving the remainder for the guests. In this file should be a line starting with "default". This determines which GRUB entry will be executed by default. Set to automatically boot to either a Xen kernel or a non-Xen kernel (needed to run VMware Server).

Install packages to support VMware Server, and update all packages.

Prepare for compilation against the kernel

Copy over the VMware Server rpm to the root home directory and install:

(Note this must be done in the non-Xen kernel). Proceed with configuration:

and agree to almost everything. On openSUSE 11 the script complained that the kernel was compiled with a different version of gcc. You must answer yes to that otherwise the script aborts immediately. We use bridged networking so that each guest is available to the external network as a stand-alone server. The bridge to select is the same as the one that was setup for Xen (br0). NAT and Host-only networking can be left out. Answer no to the question about the administrator account if you want to use root, or for security enter a non-root user to administer the machine. The guest installation files are placed in a default location which may need to be changed. We prefer to place these on separate disk drives for performance reasons.

Fedora 10

Fedora 10 has an advantage over other distributions in that it runs a 1KHz clock instead of the 250Hz clock preferred by desktop distros. A higher clock rate is preferred by VMware server. Aim for a minimal installation, which is achieved by deselecting all packages during installation, including the Base system. This leaves about 177 packages only. When installation has completed, install a few packages according to taste, such as man, emacs-nox, openssh-server and openssh-clients, wget etc.

Disable Selinux to make things easier for the moment. Edit /etc/selinux/config and change the SELINUX line to permissive. If it is changed to disable, it will not be easily reversible, but I still had trouble even with the permissive setting. There is a discussion of this on the Fedora Forums. Reboot to confirm the setting.

Add the following to /etc/sysconfig/iptables just before the first REJECT line, and reload iptables:

Prepare for VMware installation:

Copy over the VMware Server rpm to the root home directory and install:

VMware Server Tuning

Search of Internet sites shows a lot of advice about performance tuning that can be done for VMware Server. However much of that advice is drawn from a small number of earlier sites that gave no explanation of the meaning or purpose of some the suggested modifications. We shall try to summarize the knowledge that is available.

References

Fewt has hints on tuning VMWare server for performance. These changes can be made through the VI web interface, or by editing the configuration files. See also Petri, Stress-free and Alfi for additional notes.

Sanbarrow and Peter Keiser (Vmare Server 1) have information on a range of configuration parameters.

VMware FAQ Item 25 discusses performance tuning also.

Host OS and BIOS

Set the CPU speed to maximum (remove cpuspeed and set the BIOS to maximum performance).

Enable VMI

The VMI has been in the Linux kernel since 2.6.21. This is supported for certain distros using recent kernels. Ubuntu, openSUSE and Fedora are included. After creating a guest VM and before installation, set the VMI enable in the advanced tab of the configuration window. The hardware version must be version 6. If the checkbox is greyed out, the following settings could be used in the vmx file:

vmi.present = "TRUE"

Enable paravirtualization for the guest.

vmi.pciSlotNumber = "-1"

allocate the PCI slot number automatically for VMI.

Force VM Operation to Occur in Memory

The following settings attempt to force most of the VM operation to occur in memory, and to prevent swapping and sharing of memory which can use up disk I/O and CPU cycles. It really only will work well if there is plenty of memory for each VM.

On Host OS in the global configuration file /etc/vmware/config

MemTrimRate=0

Memory allocation inside the guest is faster because it doesn't take and give memory to the host OS upon all requests

sched.mem.pshare.enable="FALSE"

Guests will not share common memory blocks and VMware server will also stop comparing memory blocks.

sched.mem.maxmemctl

Maximum amount of memory that can be reclaimed from the selected virtual machine by ballooning, in megabytes (MB).

mainMem.useNamedFile="FALSE"

When allocating memory VMware will store parts of the memory in a file. This file will be the same size as the memory allocated to the guest VM. This file exists because the RAM allocation method used is mmap. By changing the setting for mainMem.useNamedFile, it will move this file from the VM's default location to /tmp. This will help if the tmpfs file system is used for /tmp. Do this by mounting /dev/shm as tmpfs and set tmpDirectory="/dev/shm" in the vmx file. When backed by /dev/shm this forces the swap to be stored in physical memory, and only dumped to the host swap file when necessary. When paired with vm.swappiness = 0 it won't use any of the server's swap unless it's out of memory.

prefvmx.minVmMemPct="100"

How the system allocates RAM for virtual machines. This setting (100%) fits all memory into the reserved RAM. Whenever possible avoid settings lower than 100%, which would allow some swapping to occur.

prefvmx.allVMMemoryLimit

How much host RAM (MB) should be reserved for all VMs.

tmpDirectory="/dev/shm"

This places the guest's temporary directory into shared memory which should have been mounted as tmpfs.

prefvmx.useRecommendedLockedMemSize=”True”

This tells VMWare whether to use a fixed sized memory chunk or balloon and shrink memory as needed. Ballooning is a technique of communicating when the host server has reclaimed some memory, by artificially grabbing guest memory to ensure that the guest is not working under a false assumption that it has more than is really there. This setting will use a fixed chunk of memory to reduce disk IO and not return unused memory to the host.

MemAllowAutoScaleDown="FALSE"

When powering on the virtual machine, automatically adjust the virtual machine's memory size if the size specified above is not available. This setting disables this.

Mem.ShareScanVM=0

This controls the rate at which the system scans memory to identify opportunities for sharing memory (default 50). This stops scanning.

/etc/vmware/config

This is the global configuration file which sets defaults for the VMs amojng other things. A summary of the above discussion in terms of additions to this file:

#-------------------------------------------------------------------------------

System Clock and Kernel Options

Redhat is preferable as the host to make use of the 1000Hz system clock compiled into the kernel. This apparently works better with VMware Server. The more common 250Hz is too small.

The kernel compiled clock rate can be found from

# grep HZ /boot/config-xxxxxxxxx-default

Add the following option to the kernel options line in /boot/grub/menu.lst to disable the tickless (dynamic clock) kernel option, and to use the deadline I/O scheduler.

nohz=off elevator=deadline

The nohz=off boot option turns off the tickless kernel option and reduces I/O constraints. Ticks are better supported by VMware.

Disk Mount Options

Filesystems such as xfs, jfs and reiserfs are better at handling very large files, and are preferable on partitions that hold the VM images. When using xfs, mount with options: noatime,nodiratime,logbufs=8, e.g. for /dev/sda5 mounted on /vm, add to /etc/fstab

/dev/sda5 /vm xfs defaults,noatime,nodiratime,logbufs=8 0 0

Add "noatime" as an option to all mounted disks in /etc/fstab (nodiratime seems to be implied but can be added also). This prevents writing the access time to each inode (which can cause I/O bottlenecks). As well as disabling atime, also set journalling filesystems to writeback by adding data=writeback to the options. This causes only the metadata to be journalled, while the actual data is not journalled. It should be a default for xfs and reiserfs. To do this for the root filesystem, check first about setting the boot options as there may be a problem getting the server to reboot.

With ext3 use the maximum 4K block size (bs=4096) as virtual disks use large blocksizes regardless . This seems to be the default.

Read ahead on the hard drive can be set to an optimal value between 16384 and 32768. This can be changed by executing

# blockdev –setra <read ahead in bytes> <device>

Host Memory Usage

Increase the default shared memory size for tmpfs to match the amount of memory in the system, so that use of RAM is maximized. For some reason openSUSE does not create a shared memory tmpfs mount. Simply add the following line to /etc/fstab:

tmpfs /dev/shm tmpfs defaults,size=4G 0 0

where the size should be the full size of the installed RAM.

The Virtual Memory subsystem (/proc/sys/vm) can be used to try to prevent swapping until memory is exhausted. In /etc/sysctl.conf add the following:

VM Tuning

• Disable the CDROM in VMs as it causes a lot of unnecessary polling activity.

• Always select SCSI as the disk type.

• Use only one virtual CPU preferably for guests. For Windows 2003 guests, 32 bit may make a positive difference.

• For each Linux guest, use boot kernel parameters (ACPI/APIC is needed for virtual SMP, but avoid this in VMware Server for performance reasons):

noapic nolapic apci=off elevator=noop clocksource=acpi_pm divider=10 nosmp

• Disable all swap files so that the host does the swapping. If the host is constrained for memory or applications that need a swap file are being used, then a swap file should be used in the guests. This allows for the use of the balloon driver and will optimize the use of RAM for the guests. However if there is plenty of RAM the guests could be tried without a swap file.

• Set a static MAC address when setting up a guest to prevent the address from being renewed when the VM is movedor copied. Suggested addresses are in the range 00:50:56:XX:YY:ZZ where XX is between 0-3F and YY, ZZ between 0-FF. Make sure that the modification is done before installing any OS.

• For each vmx file (i.e. each VM) in /usr/lib/vmware, make the changes (note some of these may already be set in /etc/vmware/config as a global setting):

VMware Server Backup/Restore

The backup solution we chose was simply to take a snapshot of each running guest VM, copy over the VM files to an external USB or eSATA drive, and then delete the snapshot. Taking a snapshot causes the virtual disk files to stop accepting any further changes, and any subsequent changes that are written to the virtual disks are written to a separate "differential" file. Deleting the snapshot re-merges the two files. For performance reasons snapshots should not be allowed to remain but should be deleted as soon as possible. Snapshots are also useful when making risky changes to the guest OS as they allow immediate backtrack to a working state. An example cron job run at night:

Note that [Windows] refers to a datastore that we have defined in VMware. Restoration of a guest can be done simply by copying back the VM files. To recover individual files from within the guest, simply mount the particular vmdk virtual disk file using the vmware-mount command.

VMware Virtual Disk Expansion

If the capacity of a virtual disk needs to be expanded, this can be done simply with the vmware-vdiskmanager tool. There is however a catch in this procedure that is often not appreciated. The expansion can be done from the web interface and also using the command line utility. In the host, change to the directory of the VM's vmdk files and use, for example:

# vmware-vdiskmanager -x 35GB Trinity-server.vmdk

This asks for the file Trinity-server.vmdk to be expanded to 35GB (where before it was less than this). Before this will work there must be at least an additional 35GB free in the host's disk partition, possibly to allow for a temporary file used in the expansion process.

If the guest is a Windows installation, Windows must also be made to see the extra disk space. In Vista and Server 2008, the disk manager tool should allow this to be done very simply. For XP and Server 2003, either a third party tool or, if the disk is not a system disk, the Microsoft diskpart.exe command line tool may be used. Note that there is a problem with this tool in regard to extended partitions that may require a hotfix to be applied. The best tool I have found is Bootit-NG by Terabyte Unlimited. This requires booting the VM from a CDROM. I couldn't get this to work correctly in VMware Server 2, but it does work in Xen. To use Microsoft's diskpart.exe on a system partition, Windows must be booted from an NTFS boot disk. Finding an iso for one of these seems to be non-trivial.

Xen

Despite being a powerful and mature hypervisor, Xen has been losing support in a number of Linux distributions. The Xen kernel is not currently available for Fedora or Ubuntu. Most Linux distributions seem to be putting their support behind KVM, which indeed may one day rival Xen but currently is a little immature. The one distribution which provides the strongest support for Xen is openSUSE. SUSE and Xen have both been developed commercially by Novell, which makes openSUSE a natural supporter of the hypervisor.

Xen is installed when openSUSE is installed as described below. Management tools for the open source version of Xen are a bit patchy but everything needing to be done can be done with what is available.

To run Xen, boot up using the Xen kernel. VMware Server will not run if it detects a Xen kernel present. Xen uses the host (dom0) as a means for each guest OS to access drivers. It also supports a number of management functions. There should be a process called xend which performs much of the management work for Xen.

References

XenSource Documentation, Novell Documentation

Virtuatopia has some advice regarding Windows 2008 guests.

TechTarget discusses block devices. There is also valuable information about libvirt drivers for Xen.

Management Tools

A straightforward way to manage Xen is to make use of virt-manager over an SSH connection. This requires X libraries to be installed on the host. If it is desired to keep the host install absolutely minimal, then a command-line approach may be used. On openSUSE, vm-install is available to create new VMs and will work in a command-line wizard mode. Then virsh or xm can be used to modify, start and stop VMs. In fact virt-manager doesn't always provide the best configuration so these latter command-line tools, along with primitive editing of configuration files, are useful even in an X environment.

Xen Guest Creation

When a new guest is created, a file is created in /etc/xen/vm with the specifications for the guest. This is only used for initial creation. Afterwards xend will take over and hold all guest status information in a database. Tools such as vm and virsh can be used to make changes to the guests, such as adding and removing hardware.

To access a GUI management tool, connect through SSH and call up virt-manager, for example:

Right-click on the top entry and select "connect". All running guests should appear, including dom0. Note: if virt-manager causes the host to freeze, use virsh to start VMs and virt-viewer to get a vnc console.

Fully virtualized guests may be created by selecting "New". This will work for paravirtualized guests such as openSUSE 11 and RHEL 5 (and Centos 5), but notably not for Fedora . Windows guests must be installed as fully virtualized. If a distribution doesn't support paravirtualization, virt-manager will complain about a missing /tmp directory.

The network should automatically be configured using the bridge. Other options may be added or modified from the entries on the summary screen. The accepted way of installing the guest OS is to use an iso image and adding a CDROM pointing to that image.

Once the guest is created it is set running and the install menu for the selected installation medium should appear. After installation, the "Details" button will lead to screens in which more hardware may be added or modified.

The "Open" button will call up a VNC window to the guest's console. This tends to be quite sluggish over any network but is useful to follow a guest bootup or shutdown.

Adding an Existing Xen Guest

If raw image files are available for an existing guest, then the procedure is very similar to the above. The "New" button is used to create a new guest, but instead of setting up a CDROM with an install image, the existing virtual disks are added. Then virt-manager will look for a bootable disk image rather than a CDROM.

Install the disks as either file type or tap:aio. Other types may be used provided the images have been setup with appropriate drivers as described below under Converting Physical Machine and VMware VM to Xen VM.

Use of virsh

This command-line tool comes in very useful for management of guest VMs. In particular an XML copy of the xend managed VM specification can be obtained from the virsh dumplxml command, e.g.:

# virsh dumpxml Trinity-server > guest.xml

Where Trinity-server is the name of the guest. This XML file can be edited and replaced with

# virsh define guest.xml

This process allows us to replace defaults imposed by the limitations of the GUI tools.

Graphical Console

The method described above to access virt-manager over an SSH connection, gave a few problems with the host freezing up. Another way to obtain a graphical console is to connect to the built-in VNC server in Xen. The defaults created by virt-manager  in a newly created VM need to be changed. Obtain the XML definition and find the entry referring to graphics, which should look something like:

<graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'/>

The port=-1 is a deprecated parameter equivalent to autoport, which automatically assigns a port number to each VM. It may be preferable to manually enter the port number (above 5900) so that you know what VM you are connecting to. The listen parameter by default limits connections to the local machine. Change to all zeros to have the VNC server listen on all interfaces. Make sure the firewall is open to these ports.

 <graphics type='vnc' port='5901' listen='0.0.0.0'/>

Install a suitable VNC viewer such as xtightvncviewer and invoke with the IP address of the host and the desired port number, e.g.

$ xtightvncviewer 192.168.1.1::5901

When using xtightvncviewer, you can only issue Ctl-Alt-Del from a pop-up that appears when the F8 key is pressed.

Conversion of a Physical Machine and VMware Server VMs to Xen

The process of moving from an existing physical machine installation to a virtual machine running on Xen, is actually quite straight-forward and reliable despite the complexity of instructions found on various Internet sites. This is a two-stage process, first moving to a VMware Server 2 VM, then to a Xen VM. In fact we can create a VM that runs alternately in both hypervisors without conversion. The use of VMware Server tools is the reason that this works so well.See Unix-Linux-Storage and Ruiz.

VMware Converter for Windows

VMware Converter is available for download from VMware. The version used was 3.03. The following instructions are given for converting a machine on which VMware Converter is running, in our case Windows 2003.

• In the conversion wizard select "Physical Computer" to convert. Select "This Local Machine". Then select the volumes to be converted. To minimize the conversion time, only select those essential to getting the VM to run. Large data disks can take a very long time to convert and may be better created a different way (a conversion may require 30 minutes or more per GB).

• To have VMware Server as destination, select "Other Virtual Machine". Give the VM a name and select a location to hold the very large files that may be created. Note that conversion will produce virtual disks files of a size that matches the amount of used capacity on a volume, not the size of the volume. Select Workstation 6 as the type (this is VM type 6, the most flexible of the VM types).

• Allow virtual disk files to expand (growable files are the smallest).

• On the networks page choose Bridged networking.

• If you are moving to Xen then don't install VMware tools, otherwise they will need to be removed later. Remove all System Checkpoints.

This will produce a set of vmdk files that should run directly under VMware Server.

Preparation of the Windows VM

• In VMware server, add the newly created guest VM to the inventory and start it. Uninstall VMware tools if necessary before any further conversion. This prevents annoyances with missing drivers and services which end up being unable to start.

• Setup the CDROM of the guest to point to an iso file for the install CD/DVD of your version of Windows. Run the batch file provided in MergeIDE which will install the necessary IDE drivers from the Windows install disk and update the Windows registry (see the 4 steps as per Microsoft KB article). MergeIDE still works despite the fact that it is written in German. You should see the word "erledigt" (done) appear twice. If the word "Bitte" appears, then it hasn't worked (edit the file to add a pause at the end). If you have a 64 bit Windows VM, then the file can be edited to point to the driver cache amd64 directory rather than the i386 directory.  The seventh line should read:

• Install the Windows gpl-pv paravirtualized drivers. These drivers are obtained from Meadowcourt. The files for fre/wnet/amd64 are the appropriate ones for a 64-bit Windows 2003 VM. Simply install these as normal, and answer yes to all questions. Windows may ask several times about installing unverified drivers for the disks.

• Shutdown the VM.

Transfer of a Windows VM to Xen

• Copy the vmdk files over to the server which has Xen installed. In our case we are using the same machine, but we need to boot into the Xen kernel eventually.

• Install the QEMU package to access tools for file format conversions:

# zypper install qemu

• Convert all VMware images with qemu-img as given below, to create raw images (which are sparse files). Other file types do not work until the appropriate virtualized drivers have been installed on the guest. We sometimes need to return to the raw image format to get the VM to boot in Xen.

# qemu-img convert Trinity-server.vmdk -O raw Trinity-server.raw

• Open up virt-manager and create a Windows VM, selecting the option "I have a disk or disk image with an installed operating system". There are a number of different choices to match the Windows installation (in our case Windows 2003 64 bit). In the summary window, select the Hard Disks link and edit the first entry. Use the Browse option to navigate to the main raw file holding the Windows installation. The default type created is "file", which will work but is no longer used. We will change this later. Add all the other disk drives in the correct order.

• In the network configuration, change the MAC address to match that in the VMware vmx definition file. This prevents a range of mysterious networking problems. Otherwise Windows tries to manage two NICs, one which is non-existent but will be configured nonetheless, causing havoc on the network.
  

• Run the VM from virt-manager. The console should show the VM booting up. Login to Windows and allow the new devices to be recognised and the gpl-pv drivers to be installed. The disk drives initially will appear as IDE drives. Reboot the VM to ensure that all driver installation has been completed. Shut down the VM.

Convert all the raw virtual disk files back to vmdk using the qemu-img utility:

# qemu-img convert -6 -s Trinity-server.raw -O vmdk Trinity-server.vmdk

The -6 forces the vmdk file to be of VMware type 6 and -s forces the use of SCSI drives. All drives may now be removed and reinstalled as virtual vmdk drives. The way to do this is to dump the VM definition to an XML file

# virsh dumpxml Trinity-server > guest.xml

Now edit the XML file to redefine the disk types. Locate the section defining the disks, which will likely have the following form:

<disk type='file' device='disk'>

This can be changed to tap:vmdk and refer to the vmdk disks created above, and the target device also changed from IDE to virtual disk.

<disk type='file' device='disk'>

Redefine the VM from the XML file:

# virsh define guest.xml

The VM should now be able to boot up in both Xen and VMware using the same vmdk files. In Xen the Windows device manager utility shows some devices that have problems with the drivers. The IDE device is not of concern since we are using SCSI virtual drives. This does not seem to affect the operation of Windows. There may be entries in the registry referring to non-existent devices. These should perhaps be left alone if VMware Server is to be used as a backup.

Transfer of a Linux VM to Xen

The procedure for converting a Linux installation is essentially the same as that for Windows. It is important that the XML file be edited to change the NIC MAC addresses to be the same as those used in the VMware guest. This ensures that the ethernet devices will be correctly recognized.

Virtual Disk Expansion

If the image file is not in vmdk format, convert it using qemu-img. Then use the procedure described above with vmware-vdiskmanager to expand the drive. Start up the VM in Xen and expand the disk drive in Windows as described above. If it is not possible to do this in VMware Server 2, then convert back to a raw image and start up in Xen. To booting from a CDROM (iso image file), use the latest version of virt-manager, which has features to simplify this process. Alternatively edit the XML configuration file and change the  line <boot dev='hd'/> to  <boot dev='cdrom'/>.

File Formats

Xen can use virtual disks based on image files, LVM volumes or disk partitions. The latter two are better performers, but file based images are best for simple backup and restore. Some useful image types that can be used in the Xen guest configuration specification are:

• phy: a physical partition, in fact any block device found in /dev. It is important to pre-partition a disk and use the partitions directly in a custom partition setup. Alternatively if an entire disk drive is used, then it should be exported to Xen as a full disk drive so that a physical partition is not partitioned again. This can provide peak performance.

• file: a file is used with the loop devices, which is a special device backed by an actual file. This is deprecated and should not be used.

• tap:aio: this uses the blktap interface with an asynchronous IO driver. This is superior to the loop device.

• tap:qcow2: makes use of the blktap interface with a QEMU copy on write file driver. The qcow2 format is similar to that used by QEMU. It allows for growable disks and snapshots (but beware).

• tap:vmdk: uses the blktap interface with a VMware vmdk open standard file driver. This provides direct compatibility with VMware guest images.

Image file types are:

• Sparse. tap:aio may not work particularly well with this as it requires additional blocks to be allocated on each write, causing a journal sync if the FS is a journalling type. However it is preferred for backup as only used storage needs to be copied to the backup medium.

• Preallocated (flat). All storage is allocated at creation, which provides better performance than a growable file type but is not so useful for backup.

Techtarget provides some information about block devices with Xen.

Instead of using disk images either a partition or LVM volume can be used. LVM may allow snapshots for backup. A raw partition may need to call on partimage and would need the VMs to be paused or shutdown.

Management Tools

Most of our remote maintenance work on the host is done using SSH with X forwarding. This requires X libraries to be present in the host.

Xen

virt-manager, a RedHat project, is a GUI tool that aims to support a number of hypervisors (notably Xen, KVM and QEMU). This polygamy can be quite confusing as each hypervisor has different characteristics and virt-manager does not properly distinguish between them in the GUI. However it does provide significant simplification of the management process.

It is recommended that the latest version of virt-manager (0.7.0 or later) be installed as it has a number of valuable additional features compared to some earlier versions, notably the addition of boot options allowing automatic boot and boot from CDROM.

vm-install is a GUI tool provided in openSUSE for setting up a VM.

qemu-img is a command line tool for converting between a number of VM file formats.

vm is a low level command line tool for managing VMs.

virsh is an alternative command line tool for managing VMs.

VMware

vmware-vdiskmanager provides a range of conversion and management options for vmdk files. In particular it permits virtual disks to be expanded or shrunk.

vmware-mount allows vmdk files to be mounted using loop devices in Linux. This is important for maintenance changes or retoration of individual guest files from backups.

Troubleshooting

1. VMware's web based utility, which is the only management interface available for VMware Server 2, is quite flaky over Internet connections and regularly logs out. It needs a measure of patience. Also in Firefox on Linux the console, which is essential in some cases to observe boot or shutdown progress and to login for initial configuration, often crashes showing the error "VMware Server unrecoverable error: (mks) Unexpected signal: 11" or shows blank. This can usually be resolved by restarting the local browser, but appears to be due to some bug in the console plugin.

2. For Xen, virt-manager over SSH seems to cause the openSUSE host to freeze. This can be annoying if you have to travel some hours just to reset the host. Sometimes working VMs will mysteriously refuse to start, giving errors like "xend.err 'Device 51712 (tap) could not be connected'" with no other explanation. The solution is usually to reboot the host, and sometimes also to redefine the VM from its XML file. It appears to happen mainly when a Windows VM is restarted, so the work-around for the moment is to shutdown a Windows VM then start it up again. Linux VMs give no problems.