Hyper-V snapshots are taking up a large amount of disk space

This has been a popular topic in the forums recently.
The scenario:  a VM is on a volume, everything is happy.
As time passes, a few snapshots are taken (for various reasons) and suddenly the disk utilization is through the roof.
What is going on?
A few things might be going on, there are a bunch of variables in the equation.
Variable 1:
How much change happened after the snapshot was taken?
Differencing disks are an overlay map, therefore you could totally re-install the VM after a snapshot and thus your VHD would be limited to the same size (the size limit of the VHD).  However, on your storage volume it would now take up twice as much space.
Variable 2:
Was the VM running when you took the snapshot?
Here is a big one.  If the VM was running the VM can be restored to that previously running state.  Thus all that occupied memory space must be saved as well.   Now, not only is the disk (potentially) using more storage, but the SQL instance in the VM was set to use 2Gb of RAM, and all of that memory space must be saved as well.
Variable 3:
What is the location of the snapshot files?
Was the default %systemroot% location changed so the snapshots reside with the VM?
The actual snapshot files can be held on any locally presented storage, they don’t have to reside with the VHD.
there are more variables, but these are the big ones that have quick and hard impact that are easy to affect.
Just some forward thinking is required.

Here is my addition from a recent forum post:
.bin files are memory snapshots - these can be as large as the configured RAM of your VM.


.avhd are the differencing disks that make up a differencing disk chain, which is the way that snapshots are implemented under the hood.

the Default behavior with R2 is that all VM files (except the configuration) are held in the same directory, where the VM virtual disks are stored (the root folder of the VM). This makes VM migration much easier and fool-proof.

The hitch with using differencing disks is that a child differencing disk can be as large as its parent. Don't think dynamic disks either. A dynamic disk has a maximum size and it is that maximum size that you must be thinking of.

This being said if you create a new VM and take the default of a 127 Gb Dynamic disk. you have the potential of consuming 127 Gb of storage. If you create a snapshot you sitll have the potential of consuming 127 Gb or storage - PLUS the storage that you already consumed withthe VHD. If you create snapshot #2 you have the potential of consuming 127 Gb of storage plus the amount of storage you consumed with the VHD and the AVHD from snapshot one. And this continues.

If you take snapshots of running VMs, then you also have the storage required to hold the memory of the running VM. Therefore, yet more storage.

MSFT finally talks snapshots in detail

Finally, after a long time of answering forums posts and my own posts here.

Ben Armstrong (or VirtualPC Guy fame) has posted a series of articles regarding snapshots and snapshot behavior.

Please, read and enjoy.  These are not going away any time soon.

Where are my snapshot files?

http://blogs.msdn.com/virtual_pc_guy/archive/2009/04/13/where-are-my-snapshot-files-hyper-v.aspx

What happens when I delete a snapshot?

http://blogs.msdn.com/virtual_pc_guy/archive/2009/04/15/what-happens-when-i-delete-a-snapshot-hyper-v.aspx

What happens when a snapshot is being merged?

http://blogs.msdn.com/virtual_pc_guy/archive/2009/04/16/what-happens-when-a-snapshot-is-being-merged-hyper-v.aspx

Why does it take so long to delete a virtual machine with snapshots?

http://blogs.msdn.com/virtual_pc_guy/archive/2009/04/20/why-does-it-take-so-long-to-delete-a-virtual-machine-with-snapshots-hyper-v.aspx

What happens if I start a virtual machine that is merging snapshot files?

http://blogs.msdn.com/virtual_pc_guy/archive/2009/04/21/what-happens-if-i-start-a-virtual-machine-that-is-merging-snapshot-files-hyper-v.aspx

Should virtual machine snapshots be used in production?

http://blogs.msdn.com/virtual_pc_guy/archive/2009/04/23/should-virtual-machine-snapshots-be-used-in-production-hyper-v.aspx

To Glom or not to Glom is a taskbar option

This has to be one of the most interestingly named Registry keys that I have come across yet.

"TaskbarGlomming"=dword:00000001

What is this?

Interestingly enough, this the key that sits behind the “Group similar taskbar buttons” settings of the taskbar.

The reason that I bring this up, is that it is an interesting statement on USA slang expressions.

Most of use are used to using the term ‘glom’ or ‘to glom’ to describe the act of ‘hanging on to’. 

Like that girlfriend you had in High School that just would not give any personal space and demanded to be hanging off of you at all times possible.

Interestingly enough, no one has entered this word into Wikipedia yet.

thefreedictionary.com has an interesting definition that fits well though:

glom - seize upon or latch onto something

Obviously when you set the Registry key “TaskbarGlomming” to true you cause all of the like programs to glom upon each other.

Just caught me as humorous today.

Migrating Debian from VMWare ESX to XenServer

Fixing Debian virtual machines after migration
After import:  Debian boots, detects new hardware, and installs new device drivers.
The boot sequence fails at the following error: ALERT! /dev/sda1 does not exist. Dropping to a shell!

This is because the source host (ESX server) presents boot devices as SCSI bus devices by default and the paths have been set within the installation of Debian.
To fix the Debian boot loader:
The Grub boot loader has dropped you into the BusyBox recovery shell. At a command prompt under initramfs.
Begin by looking at the Storage tab in XenCenter and note the presentation (device path) of the virtual disk. This describes the bus that is used to present the virtual disk to the virtual machine. This information does not appear for all virtual machines, but in this case it is available so I will use it to save time attempting to discover the interface and volumes.

This shows that the virtual disk is on /dev/hda – this represents the first IDE interface.
I will begin by returning to the virtual machine console and listing the IDE devices.

Five IDE devices have been identified.
· /hda represents the disk in position 0 IDE controller 0
· /hda1 represents the first partition of the disk /hda
I am going to make an assumption that my Debian installation is on the first partition of the IDE interface. Please note that some Linux installations place the swap on first volume, so some repetition of the follow steps might be necessary to discover the proper volume.
At the command prompt I am going to mount the partition as /root.
This is safe in this case because /root was not loaded as noted by the Alert! message identifying the missing boot device.

Now that the file system is mounted under /root I will verify that it appears to be my Linux installation

Listing the contents I see \boot, \root, \etc, \bin, and so forth. All of these are directories that I expect to find at the root of a Linux installation.
I will first fix the device descriptor file /etc/fstab.
· Change to the /root/etc directory
· Open fstab in an editor

Note the entries that point to /dev/sda; these are the entries that we will be changing to /dev/hda since a SCSI disk controller is no longer available.
Modify the /sda entries and save the file.

Similar steps need to be repeated for the Grub boot loader and the device map.
Change to the /grub folder.

Open the menu.lst file in an editor.
*Note: This file name is menu.Lst not menu.1st
Near the end of the file are the entries that we are concerned with.
Find the boot menu options that point to the /sda device.

Change the entries to /hda. Then save the file.

The virtual machine is now ready for reboot.
Simply enter ‘reboot’ at the command prompt.
During reboot, you may notice that the boot hangs after detecting the USB Tablet device. Select <Enter> to accept the default identified mouse device.
Previously under VMware, my Debian virtual machine was running X server for a graphical desktop.
As with RedHat an error that X server failed to start. After selecting No to bypass the detailed troubleshooting screens a dialog is presented that the X server has been disabled.

After selecting OK you are presented with a logon prompt.
To repair the X server:
To reconfigure the X server two methods can be used:
1) Edit the file /etc/X11/xorg.conf directly
2) Allow automated reconfiguration and accept the identified devices.
To run the automated reconfiguration:
Login as root and then execute reconfiguration for the xserver package.

Step through the wizard and accept the automatically detected defaults.
When the wizard completes start xserver by entering startx or reboot the virtual machine.

Migrating SLES from VMWare ESX to XenServer

For SuSE Linux Enterprise Server I require a “Helper” virtual machine to mount and repair the file system. This is because SLES recovery console does not include an editor.

After migrating SuSE and booting the boot loader fails at: “waiting for device /dev/sda2.” This is as expected because /sda refers to a SCSI bus and on XenServer SuSE actually sees an /hda (IDE) boot device.

The Helper VM can be created using the XenServer provided Debian Etch template virtual machine (this template includes the media, making it practically ready to go). The included Debian distribution also works with the SUSE reiserFS that is installed by default.

It is also of note that SuSE has a full Xen aware kernel and can be further optimized by presentation of the boot devices as Xen Virtual Disks and by loading a paravirtualized kernel. These optimizations are outside of this article; this is specifically focused on having a running virtual machine.

Import the VM to XenServer:

In my examples I am using XenConvert 2.0 to consume the VMware OVF virtual appliances, however Citrix Project Kensho can also be used.

Creating the Helper virtual machine:

In XenCenter select VM -> New

Choose ‘Debian Etch 4.0’ as the template (this template provides the installation template plus the operating system, nothing to download).

Name the virtual machine “HelperVM” and complete the New VM wizard accepting the defaults, allow the VM to boot, and open the console of this VM.

At the console of HelperVM enter a new root password, VNC password, and a host name (‘HelperVM’ is my suggestion).

Mount the SLES virtual disk to HelperVM:

In XenCenter select the SuSE virtual machine, and then select the Storage tab.

Select the virtual disk (make a note of the disk name) and then click Detach.

*Note: the VM must be powered off to detach a virtual disk.

Select HelperVM, then the Storage tab, and then click Attach.

Select the SuSE virtual disk from the Storage Repository and click Attach.

Return to the HelperVM console.

Note that HelperVM should have auto-mounted the volume (in this example HelperVM was running when I attached the virtual disk). My example added the controller device xvdc with the partitions of xvdc1 and xvdc2.

This can also be seen in the Storage tab of XenCenter.

Return to the console of HelperVM and create a path to mount the volume and mount the first volume.

Mkdir /mnt/suse

mount /xvdc/xvdc1 /mnt/suse

Note the error that this looks like swap. I will try to mount the other volume.

Switch to the mounted file system and list to verify that this appears to be the root volume.

Repairing the boot loader:

From this point forward the process is fundamentally no different than repairing Debian or modifying the Grub menu and fstab of any Linux distribution.

I will begin by repairing fstab.

From the root of the mounted SuSE virtual hard disk (/mnt/suse) change to the /etc directory and open the fstab file in an editor.

It should look similar to my nano editor screen below:

I am going to modify the two entries that point specifically to a SCSI presented boot device to IDE.

Previously, in the XenCenter Storage tab for the SuSE virtual machine I observed that the virtual disk was presented on an IDE controller.

The new fstab should resemble this:

Now, to proceed to the Grub boot loader menu.

One way to approach this is to copy an existing entry to a new entry and make the necessary modifications to the new entry. In this example I am modifying the existing entries for the new hypervisor.

Change to the /boot/grub directory ( cd /mnt/suse/boot/grub )

And open menu.lst in an editor.

Find the entries that refer to /dev/sdaX and change them to /dev/hdaX. In the screen shot above this is /dev/sda2

Then save the modifications.

To safely continue I need to un-mount the SuSE virtual disk from the HelperVM.

Return to the root of the file system ( cd / ) and use the umount command to un-mount the xvdc2 device.

To continue to repair the SuSE virtual machine, the virtual disk needs to be detached from the HelperVM and attached back to the SuSE virtual machine.

Mount the SLES virtual disk to the SuSE VM:

Begin by shutting down HelperVM.

Select the Storage tab of HelperVM. Select the SuSE virtual disk and select Detach.

Select the SuSE virtual machine, select the Storage tab, click Attach

Select the correct virtual disk and Attach.

Open the console of the SuSE virtual machine and power it on.

Additional repairs:

As with other Linux distributions, if X server was used to present a graphical console it will require repair due to the capabilities of new video devices.

X server is then disabled.

To repair X server, logon as root and repair the X server by running SaX2

At the command prompt execute the command sax2 -f

At the completion of the wizard X server can be started by executing startx or rebooting.

The one thing that you will notice is that there is no mouse support. Setting up VNC Server within the virtual machine and connecting to a graphical console using this VNC connection can resolve this situation.

Migrating CentOS from VMWare ESX to XenServer

Repairing CentOS virtual machines after migration

After import:  the experience of CentOS is identical to RedHat. In fact, if you watch the boot screen closely, you will notice the words “Red Hat” during the boot sequence.

I have outlined the process below.

CentOS boots, detects new hardware, and installs new device drivers. Nothing else needs to be done.

If VMware Tools was installed within the virtual machine then the following X server error will be seen.

The X Windows server will fail to start due to new video devices being detected. This is because of custom video devices and drivers that depended on the underlying VMware host. The system will prompt the user to repair the X server. If the user accepts the automated repair, X Windows will be functioning properly.

Selecting Yes presents diagnosis screens, that might be helpful for those proficient with Linux.

Selecting No skips the diagnosis screens and begins repairing XWindows.

Select Yes to begin repair.

Enter the password for the Root user account.

CentOS begins the process of hardware redetection and configuration of X Windows.

A new settings dialog box is presented.

Click OK to accept a known working configuration.

Acknowledge the information box.

Select Enter to accept the dialog that the configuration has completed.

At this time X Windows restarts and you are able to logon to the graphical desktop.

Migrating RedHat from VMware to XenServer

Fixing RedHat virtual machines after migration

After import: RedHat will boot, detect new hardware, and install new device drivers. Nothing else needs to be done.

If VMware Tools was installed within the virtual machine then the following X server error will be seen.

The X Windows server will fail to start due to new video devices being detected. This is because of custom video devices and drivers that depended on the underlying VMware host. The system will prompt the user to repair the X server. If the user accepts the automated repair, X Windows will be functioning properly.

To repair the X Server:

Selecting Yes presents diagnosis screens. Selecting No skips the diagnosis screens and goes straight to repairing XWindows.

Select Yes to begin repair.

Enter the password for the Root user account.

RedHat begins the process of detecting the hardware and devices and modifying the configuration of X windows. A new settings dialog box is presented.

Click OK to accept a known working configuration.

Acknowledge the information box.

Select <Enter> to accept the dialog that the configuration has completed.

At this time X server restarts and you are able to logon to the graphical desktop.

Hyper-V Troubleshooting and Events Guide.

Here is the Troubleshooting Guide for Hyper-V.

Like it notes at the top of the document; it isn't complete nor finished. But it is a great start.

http://technet.microsoft.com/en-us/library/dd578337.aspx

It also serves as a little peak into the architecture of Hyper-V and how the various entities are linked and ordered.
It helps make additional sense of problem situations.

Migrating VMs between Hypervisors – the hypervisor

The key items to be aware of with migration of Linux virtual machines between hypervisors is:

1) What controller does the respective hypervisor present as the boot controller

2) How are other devices emulated (video, mouse, keyboard, NIC)

3) Are virtual machine tools installed from the source hypervisor

a. Do these present custom devices or device drivers

4) Is the Linux virtual machine using a paravirtualization aware kernel.

The most critical of these considerations in the success of being able to recover a virtual machines without reinstalling is the presentation of the disk interface. The operating system can be repaired by modifying the boot devices identified in the boot loader and in the file system table.

The presentation of the boot disk can be very different between hypervisors. This is analogous to removing a hard drive from an IDE controller and attaching it to a SCSI or ATA controller and expecting the installed operating system to boot and run properly. With a windows operating system the usual result is the Blue Screen of Death (BSOD) with a 0x0000007B error. This error is generally associated with critical boot device driver not being available for the bus the disk is on.

With Linux operating systems the issue is not a critical boot device, but rather a path to the /root partition. The most common Linux boot loader is Grub. Depending on the distribution Grub has the path to the root defined in the boot loader entry. Another item used by many Linux distributions is the File system Table or fstab. Fstab is simply a file that serves as a shortcut to the location of operating system volumes such as root, swap, var, etc. All of these entries can be edited after mounting the installation volume.

The most resilient of the operating systems to migration is RedHat. The current RedHat distributions appear to use virtual points to the disk volumes. Instead of /hda for IDE 0 or /sda for SCSI 0 I notice /VolGroup00, a more generic descriptor for the first boot disk.

Virtual machine tools may have secondary impacts and errors related to supporting applications such as XWindows or VNC Server that are running within the virtual machine. There are also downstream situations where a particular set of virtual machine tools has a negative impact on a new envirnonment (ie. The VMware tools have a negative performance impact when they run within a VM hosted on Hyper-V or XenServer).

In the case of a paravirtualized kernel the result depends on the host system being migrated to. The result could be that the VM continues to function, or the VM could fail to boot with a kernel error because the new host does not support the kernel of the previous host. This is a more complicated case and will need to be considered on a case by case basis.

Differences between hypervisors:

From a user perspective all hypervisors provide similar services – they host and run virtual machines. However, the architecture and implementation of each is very different.

Focusing on how a hypervisor presents virtual disks to a virtual machine will be the remainder of this writing. If a virtual machine cannot boot after migration, then the exercise has been considered a failure.

Hyper-V

The Hyper-V model is that the booting volume of an operating system must be installed on a virtual IDE controller. For a windows virtual machine this is what we commonly refer to as the C:\ or C:\Windows volume. In the Linux world, this is where the boot loader and the root volume reside.

Therefore, when Hyper-V is the source an IDE interface can be assumed.

VMware ESX

The ESX model has always been SCSI. All disks of a virtual machine appear to the installed operating system as residing on a SCSI controller. Historically, this presented installation challenges as some operating systems, such as Windows XP and Windows 2003, did not include a compatible SCSI driver.

Only in vSphere 4 is this model changed, to allow the option of IDE. However, in the Release Candidate the default controller is still SCSI.

XenServer

In the XenServer world the disk devices can begin to appear very strange. XenServer virtual machines can see disk devices are IDE, SCSI, or a SCSI-like controller. Much of this behavior is defined by the template that was used to create the virtual machine combined with paravirtualization awareness of the OS.

A Windows operating system will behave as if the boot device is IDE. However a Linux operating system can identify a variety of devices such as /hda, /sda, /xvda, and /QEMU.

Moving virtual Machines in practice

In practice a Windows virtual machine will move between XenServer and Hyper-V rather easily. There is usually the recognition of new devices on the receiving platform, but generally nothing that cannot be repaired, nor prevents booting the operating system.

From or to VMware was the historic problem because of the need to change from a SCSI controller to something else. For Windows operating systems this requires the injection of a critical boot device driver that supports the new platform.

Never resize a VHD with snapshots or differencing disks – more about differencing disks

I keep seeing this item coming up in the forums over and over again.

For some reason I need to make the virtual disk of my VM larger (maybe I downloaded too many patches, maybe too many temp files, maybe too many log files).

The reason does not really matter.

So I open the Edit Disk wizard, I find the VHD and I Expand the virtual disk.

Excellent, now my VM has more space and I go into the operating system and diskpart and perform the actions that I need to for the OS to use the additional space.

This works all to well when all you have is a single VHD.  One VHD, no snapshots, no differencing disks.

I have written about snapshots quite a bit over the past year.

A snapshot is a moment in time.  The technology that Hyper-V uses to achieve this is a Differencing Disk.

The differencing disk is a child disk that is linked to its parent.  The easiest way to describe the concept of the differencing disk is to think of overlays.

In the era of the small hard disk we used to think about sectors and blocks.  But things have changed, and most folks don’t concern themselves with that anymore – a disk is a place for stuff and that place has a capacity and that is all we really concern ourselves with.  It is the blocks and sectors that form the map that we need to be aware of.  Think of it like a dart board, but wil ar more rings.

The root VHD is the map of a disk.  A digital representation of a physical hard disk.  It has blocks, sectors, etc.

The differencing disk is also a map just like its parent, however, it only contains the blocks and sectors of its parent that are different, than the parent.

To begin my analogy, I am going to think of the single VHD as a map or drawing.

Adding a differencing disk (taking a snapshot) is like laying a piece of tracing paper over a drawing, and only coloring in some features (the changes from the point the snapshot was made). 

The drawing does not change, only the way the empty places are colored in.

If you add a second differencing disk (now a chain of three virtual disks – a grandparent VHD, a parent differencing disk – a child differencing disk).  This lays a second piece of tracing paper over the drawing, and we proceed to change how the empty spaces are colored in.

What happens when you resize the VHD?

When you resize the VHD, the disk map or drawing, is changed.  I have a different combination of sectors and blocks.

Now, the spaces that I colored in on my tracing paper (differencing disks) no longer aligns with the empty spaces in the drawing.  The result is a really ugly picture.

The result with my VM is that I just broke the chain.  And I broke it right at the root.

There is currently no method (that I know of) that will allow resizing of a virtual disk that contains snapshots – differencing disks.

If you need to resize the virtual disk of a VM and there are snapshots then the snapshots must be deleted (and the VM turned off, then wait patiently).  This has the end effect of taking each piece of tracing paper and applying its colors onto the layer underneath.   The term is called merging.

The layers of tracing paper are applied back to the primary drawing, the root VHD.

In the end, you have a single VHD that you can safely resize.

Is this the only way you can end up with a single VHD?  No I can think of at least a couple more creative ways to end up with a single VHD for resizing.

The key is that you cannot resize a virtual disk with snapshots or differencing disks.

I hope that someone that is thinking about doing this finds this article in Google or Live Search and saves themselves some big recovery pain.

Migrating VMs between hypervisors – the Theory

Lately, I have been spending an incredible amount of time migrating virtual machines between hypervisors.

There are some issues that we all need to consider. Most V2V tools do this for us, they at least make sure that the VM is in a condition that it can boot. However, what are the issues that really prevent simply copying virtual disks between platforms?
The very first issue that needs to be considered is the difference between the hypervisors themselves.

For this writing I am going to focus on how the hypervisor presents devices to the virtual machines; in particular, the boot device for the virtual disk.

The primary consideration for success when modifying the hardware under an operating system is always: will it boot after I make the modifications? In this case, will the VM boot after I move it from hypervisor A to hypervisor B.

After that, there are secondary considerations; am I able to access the VM console? Are the VM tools from the previous platform incompatible or negatively impacting performance on the new platform?

In the simplest way of thinking about this, these questions are no different than when replacing hardware on a physical PC and dealing with how the installed operating system responds.

Most of us have dealt with that scenario at some point in our career as computer geeks.

However, in this case we may be tempted to think that since things are virtual, they should just work. Hypervisors differ in many aspects. In this case, the presentation of the boot disk can be very different between platforms.

This would be analogous to removing a hard drive from an IDE controller and attaching it to a SCSI or ATA controller and just expecting everything to work.

With a windows operating system the usual result is the dreaded Blue Screen of Death (BSOD) with a 0x0000007B error. This can generally be considered a critical boot device driver problem. The operating system simply says: “whoa, where did my files go? I expected them on this controller, in this place and there is nothing there.”

It is in this place where conversion tools reign supreme.

At the same time, we might be in a situation where we don’t have a conversion tool, or can’t afford one. How do we deal with that.

In the next installment, I will take a look at three hypervisors: XenServer, Hyper-V, and VMware and look at how they present disk devices to virtual machines.