Provisioning Server (PVS) Cache on device issues

I recently ran into some issues streaming VMs and taking advantage of the local storage on the hypervisors for the local cache.

Many of these issues were due to my lack of experience with the product, so I am making notes here for future use.

First, if you are not familiar with Provisioning Server or have not taken a look at it for a few revisions, go download the latest version.  There have been a lot of improvements since the last time I looked at it (a few years ago). 

Cache on Device Hard Drive is actually from the PVS feature to stream to bare metal.  In the VM world it allows me to use local storage that is relatively inexpensive (unless you use SSDs – but hey, just another possibility) and just fine when you don’t want to persist the cache disk.

Okay, enough of that, back to my cache disk issue notes.
Firstly, when you create your Template VM, be sure to create the local cache disk as its local drive.

Secondly, don’t forget to format this disk for your streamed OS.  So, for Windows, format it NTFS (I am just booting the VM that will be my template to WinPE and using DiskPart to avoid ACLs.)
Third, make that cache disk large enough.  This was my most perplexing puzzle and I fiddled for a couple days with this off and on.

When I booted the VM and login I would get this strange error message:  “Windows created a temporary paging file on your computer”

This issue is specific to using the Cache on Device setting when your vDisk is set to shared mode.  The root is that the default page file location cannot be written to as it is the shared vDisk image.  The ancillary detail is that the page file gets moved to the device cache disk which is some drive other than %system%.

So, I had my local cache, all formatted, as small as it thought it needed to be.  I booted my vDisk and the error.  I set the vDisk to private mode and fiddled around with paging file settings.  I went back to my template VM, booted in shared mode, no paging file. 

I searched and found all kinds of related errors; moving it off C:, using a SCSI attached VHD instead of IDE (I happen to be using XenServer so this was not it), ACLs on files within the VHD, also obscure errors where the programs are expecting specific folders to exist on the cache disk.  All of these are totally valid errors and entirely possible in their scenario, just not mine.

Very frustrating.  It was when I booted with the VM template in private mode with the cache disk attached and I attempted to move the page file that I got a useful error:  the destination is not large enough.

Duh!  My device cache disk (the virtual disk of the VM) was not large enough for the caching of temporary files AND the page file of the VM operating system.  So silly.

I went back and recreated my template VM, with a larger cache disk – taking into account any dynamic memory settings.

Once I made this larger VHD (formatted with Diskpart) and booted using private image mode with cache on device hard drive all was good.

TCP Checksum Offload is not equal to TCP Task Offload

This is an issue that lately I have been answering a lot in the Hyper-V TechNet forum. 

Folks find a link that refers to disabling Checksum Offloading or TCP Offload  to help with strange networking behavior with an application server (Remote Desktop Services, XenApp, Exchange, SQL, SharePoint, etc.).  So they disable “TCP Checksum Offload” using netsh on the Hyper-V Server.  The end result is that the problem is not resolved, so they come to the forum.

In actuality they did not disable the correct feature.  So here is my third attempt to try and straighten this out.

In this particular scenario the features (yes, multiple with some NIC drivers) are referred to as TCP Task Offload.  These are things such as TCP Checksum Offload, Large Send Offload, etc.  And they are properties of the NIC driver – it is not a single server level setting.

There is a really ancient Microsoft KB article that talks about this.  You want to use Method 3 in this article:  http://support.microsoft.com/kb/888750

I am going to paraphrase it here:

Method 3
If you do not want to disable TCP segmentation offloading on the whole system, and you want to only disable TCP segmentation offloading on the network adapters that Virtual Server 2005 guests use, you must not add the DisableTaskOffload registry entry that is described in Method 2. Instead, you can disable the task offload properties on the Advanced tab of the Properties dialog box of the network adapter.

Warning When you disable the task offload properties, guest virtual machines that are attached to the same virtual network may temporarily disconnect from the virtual network.

To disable the task offload properties, follow these steps:

1. Click Start, click Run, type ncpa.cpl, and then click OK.
2. Right-click your network adapter, and then click Properties.
3. Click the General tab, and then click Configure.
4. Click the Advanced tab.
5. In the Property box, click the Offload TCP Segmentation property.
6. In the Value list, click Off, and then click OK.
7. If you also have the following task offload properties in the Property box, you must repeat step 5 to step 6 to disable these properties:
• Offload Receive IP Checksum
• Offload Receive TCP Checksum
• Offload Transmit IP Checksum
• Offload Transmit TCP Checksum
  

If you have Server Core the following might help:

http://social.technet.microsoft.com/forums/en-US/winservercore/thread/d0c55df9-a27c-4876-bc5a-8ac7f1b46462/

“There are some settings for TCP/IP, go to "netsh interface tcp" and then run "set global" and you'll see all the options for some of the advanced TCP/IP configuration. One of those may help.”

The magical registry settings are documented in this MSDN article: Using Registry Values to Enable and Disable Task Offloading:   http://msdn.microsoft.com/en-us/library/ff571012.aspx

SCVMM 2012 XenServer demonstration tricks

This is a little trick that I picked up from an unnamed MSFT person.

Say that you want to have a portable demonstration environment for SCVMM 2012 to show off all of the bells and whistles.  You have a limited amount of hardware, say two servers. (maybe even one).

You can install Hyper-V.  And you can install SCVMM into a VM.  That is all fine and dandy.

You can then install Hyper-V Server into a VM as well.  And this looks good until you try to do something that requires the VM within the virtual Hyper-V to power on.  Then it falls apart.

Well…  you can mix your hypervisors up.

You could install a XenServer within a Hyper-V VM and even create a pool using two or more.  They will join and become a Resource Pool.

The key is that you give the XenServer VM a Legacy Network Adapter for installation to proceed.

You can then deploy VMs to the XenServers using SCVMM 2012 (or XenCenter if you like).  And the VMs will boot.  (yes, all disclaimers apply – a hypervisor in a VM is not a place where you expect anything to be fast, but it can show that something works).

There is also a sneak here as well.  Windows VMs will not boot – the reason is that a Windows VM requires an HVM type VM, which requires that the hypervisor in the VM must own the hardware – and it doesn’t.  Linux VMs will boot just fine.  As they can exist as PV VM’s in XenServer or as HVM type VMs.  It is the PV VMs that will boot.

Now, you can always use Hyper-V and install that into a VM running on Hyper-V.  I know folks that do that.  And these can be managed as well – but none of the VMs can be powered on.  So if you need to test SCVMM 2012 and you need to show VM power operations, you need to use Linux VMs running in a virtual XenServer.

SCVMM 2012 beta with XenServer support

The SCVMM 2012 beta was announced at MMS.  Go pick it up here:  http://www.microsoft.com/downloads/en/details.aspx?FamilyID=e0fbb298-8f02-47e7-88be-0614bc44ee32

Now, I will warn you.  This is a big update. Not just some little incremental update.  It includes some new ways of thinking about your data center.  And it includes a lot of “cloud” type words and ways of thinking.

I consider this update a paradigm shift in the VMM world.  A big change in thinking, automation, abstraction, packaging, and delivering.

It also includes support for managing XenServer.  This is a feature that is near and dear to me.  There is a supplimental pack that is required for your XenServers which can be obtained from here: http://www.citrix.com/xenserver/microsoft-beta

This gives the ability to deploy too and manage XenServer.  You still need to setup your environment using xe or XenCenter (no different than Hyper-V) – but you can control and manage VMs with SCVMM.

Other MSFT folks have mentioned the other big paradigm changes, so I won’t cover them here.  http://blogs.technet.com/b/scvmm/archive/2011/03/22/system-center-virtual-machine-2012-beta-available-now.aspx

WSMAN Namespace Handling in PowerShell

For some time now I have been working on handling XML with PowerShell – not XML that I make mind you, that appears to be relatively easy as the plethora of examples out there keeps showing me.

I am handling XML that I get back as a blob from a call to a WS-MAN provider.  It has Namespaces – that changes the game big time.

The best general reference I have found is Dr. Tobias Weltner (he is the brilliant person behind PowerShellPlus – which is an IDE that I simply don’t know how people write complex PowerShell scripts without).  This article; http://powershell.com/cs/blogs/ebook/archive/2009/03/30/chapter-14-xml.aspx talks about XML and PowerShell, but it misses the one thing that I needed, Namespace handling. 

A bit of digging let me to a C# article about xpath and xml namespaces – that sent me to the real tidbit I needed Select-Xml; http://technet.microsoft.com/en-us/library/dd347617.aspx

First I needed to workout what my namespace selection problem really was.  Here is the mess that I get back:

<n1:SomeCimMethod_OUTPUT xmlns:n1=http://schemas.someone.com/wbem/wscim/1/cim-schema/2/SomeCimClass xmlns:wsa=http://schemas.xmlsoap.org/ws/2004/08/addressing xmlns:wsman=http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd xmlns="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xml:lang=""> <n1:ThingOne><wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:Address> <wsa:ReferenceParameters><wsman:ResourceURI>http://schemas.someone.com/wbem/wscim/1/cim-schema/2/Image</wsman:ResourceURI><wsman:SelectorSet><wsman:Selector name="__cimnamespace">root/cimv2</wsman:Selector><wsman:Selector Name="CreationClassName">Image</wsman:Selector><wsman:Selector Name="ID">2c8ba04e-53b8-504d-f616-061a43bb46bf/969f4a72-4a0d-4044-b41e-f3025377d067</wsman:Selector><wsman:Selector Name="CreationClassName">Creator</wsman:Selector><wsman:Selector Name="Name">2c8ba04e-53b8-504d-f616-061a43bb46bf</wsman:Selector></wsman:SelectorSet></wsa:ReferenceParameters></n1:ThingOne><n1:ThingTwo>57702fd0-9e92-43dc-9ac6-537719b73473</n1:ThingTwo><n1:ThingThree>4e4449df-8710-4358-8290-44d7b4264d46=403ef95b-0309-417e-86d8-c75066439419,c735019c-2198-4d53-a6ac-668d38e6a81d=eb15c741-5a05-4377-91b5-7bd95ab21f3d,2b2ad08b-ecdf-42de-9f03-1050862b99fb=e2aae65c-dd64-49f3-a796-e12fecdc2b46,97c47f43-55af-438f-83b9-2d4a01733ce7=fff39bf0-9d21-4475-b7c7-9e96eb35e8d8,ee8e54e2-b499-438f-a62f-67c024e5921a=ebc63996-6399-4ffd-a5ad-6bd0dcf2036f,6fa65d8c-7cbb-438f-a2ea-35e498c525c5=ae960929-4cd6-42b3-9159-f4e0119cae92,80855f0f-1e22-44bf-892c-c8ca1fd7af59=30dd2807-5566-4655-822a-4f6780f0fdaa,57702fd0-9e92-43dc-9ac6-537719b73473=969f4a72-4a0d-4044-b41e-f3025377d067</n1:ThingThree><n1:ThingFour>57702fd0-9e92-43dc-9ac6-537719b73473</n1:ThingFour><n1:ReturnValue>0</n1:ReturnValue></n1:SomeCimMethod_OUTPUT>

If you look into this blob (there is a good reason developers call these blobs) you will see that each element is preceeded by the namespace “n1”.  Howerver, if you simply cast this to $blob = [xml]$blob it looks entirely different and you don’t really realize that each element is part of namespace “n1”.

PS > $blob.SomeCimMethod_OUTPUT

n1            : http://schemas.someone.com/wbem/wscim/1/cim-schema/2/SomeCimMethod
wsa           : http://schemas.xmlsoap.org/ws/2004/08/addressing
wsman         : http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
xmlns         : http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
lang          :
ThingOne  : ThingOne
ThingTwo     : 57702fd0-9e92-43dc-9ac6-537719b73473
ThingThree  : 4e4449df-8710-4358-8290-44d7b4264d46=403ef95b-0309-417e-86d8-c75066439419,c735019c-2198-4d53-a6ac-668d38e6a81d=eb15c741-5a05-4377-91b5-7bd95ab21f3d,2b2ad08b-ecdf-42de-9f03-1050862b99fb=e2aae65c-dd64-49f3-a796-e12fecdc2b46,97c47f43-55af-438f-83b9-2d4a01733ce7=fff39bf0-9d21-4475-b7c7-9e96eb35e8d8,ee8e54e2-b499-438f-a62f-67c024e5921a=ebc63996-6399-4ffd-a5ad-6bd0dcf2036f,6fa65d8c-7cbb-438f-a2ea-35e498c525c5=ae960929-4cd6-42b3-9159-f4e0119cae92,80855f0f-1e22-44bf-892c-c8ca1fd7af59=30dd2807-5566-4655-822a-4f6780f0fdaa,57702fd0-9e92-43dc-9ac6-537719b73473=969f4a72-4a0d-4044-b41e-f3025377d067
ThingFour : 57702fd0-9e92-43dc-9ac6-537719b73473
ReturnValue   : 0

In my example I am looking for the element “ThingTwo” which is really “n1:ThingTwo”.  The detail is that it exists within namespace “n1” and because of that $blob.SelectNodes and $blob.SelectSingleNode were totally failing me.

So, how do I find a single element within this?

First, my $blob has to be an XML document, in this case the return from the WS-MAN provider is all formatted properly, I just need to cast it to an XML document (as in PowerShell everything is a generic type of Object by default).

$blob = [xml]$blob

$blob.GetType() should return “XmlDocument” as the Name.

Then i have to make the XML parser aware of the namespace and pass that into the Select-Xml method.

$namespace = @{n1=http://schemas.someone.com/wbem/wscim/1/cim-schema/2/SomeCimMethod}

Now I can use Select-Xml to find my element.

Select-Xml -Xml $blob -Xpath "//n1:ThingTwo" -Namespace $namespace

Importing the XenApp EVA is really this easy.

I’m going to import the XenApp 6 Evaluation Virtual Appliance into Citrix XenServer 5.6 using a new feature of XenServer 5.6 and XenCenter 5.6 called the Disk Image Import Wizard.

You can also watch this in action here: http://www.citrix.com/tv/#videos/2368

I will assume that you have already downloaded and installed XenServer 5.6 and XenCenter 5.6.

First, go to the download website for the Citrix XenApp EVA. http://www.Citrix.com/tryxenapp

A login with a MyCitrix user account is required. Be sure to download two items: the Getting Started Guide and the Evaluation Virtual Appliance package for XenApp 6.

After downloading the Evaluation Virtual Appliance, execute the CitrixXA6EVA.exe to expand the self extracting archive into a folder that contains a VHD and licensing documents.

Now, open (and read) the getting started guide. The import wizard requires some important information regarding the configuration of the virtual machine. On page four, the guide states that the EVA requires at least 2GB of RAM and the VHD requires at least 30GB of storage space. So make sure that your XenServer has a Storage Repository with at least 30GB of available space.

Open XenCenter and connect to a XenServer 5.6 host (using version 5.6 is required). In the left hand tree view select the host and then right click for the context menu and choose the Disk Image Import.

In the import wizard browse to and select the VHD and choose next.

Enter the information for the virtual machine. The VM needs a name, the number of virtual processors to assign and an amount of memory.

Note: I am unselecting the Run Operating System Fixups option because I know that this VM contains Server 2008 R2 and that it was built for Hyper-V. Server 2008 R2 can handle hardware changes pretty well (single to multiple processors, critical boot devices, etc.). XenServer also presents a critical boot device controller similar to the Hyper-V IDE specific boot device.

Then begin the import.

Once the import completes, select done.

At this time: power on the virtual machine and login at the logon screen. The domain and user account and password are in the Getting Started Guide.

If the screen of the VM flickers, Windows Server 2008 R2 is busy setting up some of the new hardware devices that it is detecting. You should still be able to login and complete the setup of the Evaluation Virtual Appliance environment.

If you would like to optimize the virtual machine for XenServer, install the XenServer tools within the virtual machine.

There is no “smoke and mirrors” involved here, it is literally this easy using the Disk Image Import wizard.

Creating a Windows 7 or Server 2008 R2 image for VM deployment

Creating a Windows 7 image for deployment to a virtual machine is not as straightforward as you might think. If you simply perform a standard installation, image that to a WIM with ImageX and then attempt to deploy that image, you will be left with a system that requires that the boot manager be recreated.

Just in case anyone is confused as to why the boot partition is there, it's so that you can use bitlocker to encrypt your system drive on the fly.  Bitlocker needs the unencrypted boot partition to work.

If you create a VM and insert your Windows 7 media and proceed to click Next through the entire installation wizard you will end up with a system that is actually installed on two partitions and not deployed back from WIM without requiring repair.

One tool that can be used is the Wim2Vhd tool. This is useful if you meet the client requirements and have a way to import the resulting VHD into a new VM. It performs the required repair for you.

However, I would like to avoid the situation of needing to perform the BCDBoot repair all together.

Say that you want to use ImageX (from the Windows Automated Installation Kit ) to image the Windows 7 installation to a WIM and then deploy that Windows 7 image to other virtual systems. I do not want to repair each and every time – I want to build the reference system in a way that this is not required.

The key is to prevent the default Windows 7 installation behavior of creating a System Reserved partition (which contains the necessary boot loader files) and a separate partition where the OS is installed (where there are no boot loader files).

Method One – pre-partition the virtual disk:

This is the most direct approach; in the end nothing looks unusual.

1. Create the VM from a template

2. Boot to WinPE (or mount the disk to another (helper) VM)

(An alternative to this is to boot using the installation media.  At the "Install Windows" screen (the first screen of the installer) type Shift+F10 to open a command prompt)

3. Run diskpart

a. List disk

b. Select the disk

c. Create a primary partition (you must use the entire volume)

d. Make the partition Active

e. Format the volume

f. Exit

4. Power off the VM ( or detach the virtual disk from the helper vm)

5. Attach the Windows 7 installation ISO

6. Boot to the ISO

7. Begin the installation wizard

8. Select a “Custom” installation

9. At the “Where do you want to install Windows” screen accept the volume that was previously formatted

10. Complete the wizard

11. Apply all installations and patches to your VM

12. Prepare your VM with sysprep

13. Boot the VM to WinPE

14. Create your VM image using ImageX.

Method Two – confuse the installation wizard:

I could not think of a better title for this, because really what you are doing is messing with the options allowed in the installation wizard until you get the behavior you want.

1. Create the VM from a template

2. Attach the Windows 7 installation ISO

3. Boot to the ISO

4. Begin the installation wizard

5. Select a “Custom” installation

6. At the “Where do you want to install Windows” screen

a. Select the virtual disk

b. Select “Drive options (advanced)”

c. Select “New”

d. The default should be the entire volume – Select “Apply”

e. Acknowledge the warning

f. Select and Delete Partition 2

g. Acknowledge the warning

h. Select and Extend the System Reserved partition

i. The default should be the entire volume – Select “Apply”

j. Acknowledge the warning

k. Select Next

7. Complete the wizard

8. Apply all installations and patches to your VM

9. Prepare your VM with sysprep

10. Boot the VM to WinPE

11. Create your VM image using ImageX.

The primary difference between the results of the two methods is:

· Method Two results in a deployed image where the C: volume name is “System Reserved”

Method One:	Method Two:

BOOTMGR is missing - Repairing a Windows 7 or 2008 R2 image after VM deployment

Deploying a Windows 7 image to a virtual machine is not as straightforward as you might think.
The primary issue resides in how the reference Windows 7 system is installed.
If you create a VM and insert your Windows 7 media and proceed to click Next through the entire installation wizard, you will have a working Windows 7 installation, that can be templated, it can be copied, you can do just about anything you like to it – as long as all copies involve the entire virtual disk.
Say that you want to use ImageX to image the Windows 7 installation and then deploy that Windows 7 image to other virtual systems.
If all that you did was click Next through the installation wizard or you did not customize your unattend.xml at all – you actually have a VM that has been installed with two volumes – a System Reserved volume and the volume where the actual system is installed.
The standard process is that you boot your system into the WinPE environment and you use ImageX /capture to create a WIM from a particular volume. Note the word “volume.” ImageX is a volume based tool, not a disk based tool.
The problem comes when we try to deploy this image. We boot into ImageX and /apply – then we reboot the virtual machine and we end up with the error: “BOOTMGR is missing”

The boot manager resides on the System Reserved volume, that is the extra partition that is created but was not captured by ImageX.
According to Microsoft TechNet documentation How to Perform Common Deployment Tasks with Virtual Hard Disks we need to use BCDBoot.exe to “configure the boot entry in the BCD store to be on the volume inside the VHD.” If you follow the link, the portion that we are concerned with is: “Prepare a VHD image to boot inside a virtual machine.”
To repair the VM where the image we need some type of recovery environment. We can get to a prompt to fix the applied image by any of the following ways: Boot to a WinPE 3.0 ISO, Boot to a Recovery Console using the OS installation media, attach the virtual disk to another known working Windows virtual machine.
For my example I have used the Windows Automated Installation Kit and created a WinPE 3 ISO. However, in testing I first simply attached the failing virtual disk to a working VM and performed the same commands. I also found references that booting into the Recovery Console of the installation media also works, but I did not test this. It is important to note that if you use WinPE the bit-ness of your WinPE image must match the bit-ness of the OS in the VM (32-bit to 32-bit / 64-bit to 64-bit).
The repair process:
1. Boot the virtual machine into WinPE (or attach the virtual disk to a running VM).

(An alternative to this is to boot using the installation media.  At the "Install Windows" screen (the first screen of the installer) type Shift+F10 to open a command prompt)

2. Discover the drive letter assigned to the virtual disk that is failing to boot.

3. Execute the following command (pay attention to use the correct drive letter – in my example “C:” is the letter that was assigned to the virtual disk, the WinPE system volume is “X:”).
C:\windows\system32\bcdboot C:\windows /s C:

4. Simply type “exit” to cause WinPE to shutdown and reboot into the VM. If the virtual disk was mounted in another VM, then power off the VM and detach the virtual disk before booting the imported VM.

FreeNAS as a storage target for simple testing

Working with various hypervisors I frequently have a need for various types of storage volumes.

We can always use local storage for testing, but how do you handle a cluster of Hyper-V Server or a Pool of XenServer hosts?

I have used a variety of services.  Right now in the lab I am running Windows Storage Server 2008 to present NFS volumes and StarWind to present iSCSI volumes (I found StarWind iSCSI far easier to get working with XenServer than the iSCSI target add-on to Storage Server).

Mind you, I don’t need any of the enterprise bells and whistles that Storage Server gives me, I simply need storage.

Just the other day someone pointed me to FreeNAS.  Hey, a storage system that can present many different types of volumes.  FreeNAS is built upon FreeBSD and is bundled as a LiveCD ISO that can be installed to a hard disk, USB, or run as a LiveCD.

FreeNAS can do NFS, SMB, iSCSI, iTunes, FTP, TFTP, SSH, UPnP, BitTorrent, and other sharing protocols – from that standpoint it is pretty slick.

I must say that other than dealing with having to learn the UI – I have been pleasantly surprised at how well it is working.

Mind you, I am running FreeNAS in a VM.  That VM is hosted by Hyper-V.  And I have a XenServer that is accessing the LUNs being presented by the NAS device.

At the moment I have an iSCSI LUN and an NFS volume presented.  the big limitation that I currently have is that I can only have 4 IDE devices on my Hyper-V VM that limits me to one virtual disk for install, and three for serving as storage LUNs.

My VM simply runs with 512 Mb of RAM and a Legacy NIC.  The rest involved about an hour of figuring out how to install and configure the LUNs. 

Very little overhead and the storage actually ran pretty well when deploying a XenServer VM to the NFS share.  I really can’t argue.

When it is time to upgrade my current lab test storage server I think I will have to boot into the LiveCD and see if FreeNAS can identify the really old adaptec SCSI card and Compaq storage array.  If it can, it will win moving forward as Server 2008 R2 no longer has drivers for my really old Adaptec SCSI card – making the antique array useless. 

But that array is not useless, I just wouldn’t trust it for anything important.

The next step might be trying to build it into a paravirtualized VM for use on XenServer.  Since it sees the XenServer virtual devices as QEMU devices, I know that the kernel is Xen aware…

Hypervisor virtualization basics a visual representation

For all of you that want a place to point folks to describe the basics of virtualization, I have put together a few videos to describe the hypervisor, CPU, and network concepts in a visual way.

The intent is to give a quick amount of conceptual information to those folks that suddenly are dealing with VMs, but might not have the experience to fully understand what they are looking at.

Hypervisor Basics:

The basics of what a full (type 1) hypervisor is.

 

The hypervisor pool of resources – the CPU:

The basics of CPU scheduling. It is far more complex than this and there are many methods.  It gets really messy when hyper-threading is introduced.

More CPU scheduling details are over at the Xen.org site (the Xen folks are more open in discussing the gritty details of all this):  http://wiki.xen.org/xenwiki/CreditScheduler

The hypervisor pool of resource – the network:

This is about virtual networks / virtual switches / bridging.  The concepts, as each vendor implementation offers different features.

Datamation puts their angle on the big three virtualization vendors

Datamation has put their spin and opinion into the world of virtualization and the virtualization engines and offerings in this comparison of Citrix (XenServer), Microsoft (Hyper-V, SCVMM), and VMware (ESX, vCenter).

Mind you, this is a judgment article – But it mentions Kensho – a project very near to my personal efforts, and DocFinder – another product I was involved in the infancy of.

You can find the article here:

http://itmanagement.earthweb.com/features/article.php/12297_3853716_1/Virtual-Server-Comparison-Xen-vs-Microsoft-vs-VMware-2010.htm

Enabling Citrix Merchandising Server paravirtualized vm to run on Hyper-V

Be warned, Citrix Support will not support your Merchandising Server if you follow these steps.

And - this is really antique and no longer works with Hyper-V 2012.

Here is my scenario:

I have a fully paravirtualized Linux virtual machine (Merchandising Server) that is made to run on a xen hypervisor (the family of hypervisors that enable Linux VMs to boot kernel-xen and run in a truly paravirtualized and highly efficient way).
I want to move that vm over to another hypervisor (non-xen) for a client demonstration.
Warning:  This might seem a bit convoluted, but it really isn’t difficult.  However there are a few tools involved.  The steps that apply to your situation all depends on how easy it is to get your virtual disk into a format that you can mount and edit the boot volume.  Follow these instructions at your own risk, a positive outcome is not guaranteed.  The resulting VM will most likely not be supported if it has problems.
Collecting the elements:
I am going to use a few things that I need to obtain (download) and tools that need to be installed.

• The Citrix Merchandising Server Virtual Appliance
• The Hyper-V Linux Integration Components ISO (v1, not R2)
  
  • Try this if the above is dead.
• Citrix Project Kensho OFT Tool
• Citrix XenConvert
• Some Linux distribution “Live CD” (i use Debian Live)
• A Hyper-V Host

Why so many tools?

The Merchandising Server is the appliance that I am using for the example.  XenConvert is to convert the XVA (XenServer export format) into an OVF based appliance.  The Kensho OVF Tool is to import the OVF into Hyper-V.  The Linux distribution “Live CD” is to mount the virtual disk of the example appliance so we can modify the Grub boot loader and drop a file.  And the v1 Linux Integration Components have the magic PV kernel shim that we need.

Details, details, missing details.  What gives?

I am going to state right now that; I am not going to go into deep and gory details describing each and every click that is required with each tool.  If you read my blog, I assume a couple things: that you have a clue, or you want one.  And that you are not afraid of figuring things out, or trying and failing (I am also implying that you know how to make back-ups of things before you go mucking them up).

The business at hand:

1. Expand the Merchandizing bz2 zip archive. (WinRAR can do this, and others as well)
2. Use Citrix XenConvert 2.x to convert from “Xen Virtual Appliance” to “Open Virtualization Format (OVF) Package”
   
   • Do not create an OVA, just an OVF.
3. Use Citrix Kensho OVF Tool to Import the OVF Package from step 2 to a Hyper-V host.
   
   • Or you could just copy the VHD from step 2 to your Hyper-V host and create a new VM.
   • Do not boot the VM at this time.
4. Attach the Live CD ISO to the VM
5. Set the boot order to boot from DVD first
6. Remove the default Network Adapter and add a Legacy Network Adapter
7. Add a second DVD drive
8. Attach the Hyper-V (v1) Linux IC ISO to the second DVD drive of the VM
9. Boot the VM into the Live CD and log in to its console
   
   • Debian will auto logon as ‘user’.
10. switch to root:  sudo –i
    
    • This is specific to Debian Live
11. Discover the IDE disks:
    
    • cd /dev
    • ls hd*
12. Mount the virtual disk (the vhd)
    
    • make a mount point folder:  mkdir /mnt/mine
    • mount the disk to the folder:  mount /dev/hda1 /mnt/mine
13. explore the volume
    
    • cd /mnt/mine
    • ls
    • Mine looks like a /boot volume:
    • 
14. Mount the Linux IC DVD drive (mine is the second dvd on controller 2):
    
    • mkdir /mnt/cdrom
    • mount /dev/hdd /mnt/cdrom
15. Copy the kernel shims from the ISO to the virtual disk
    
    • cd /mnt/cdrom/shim
    • cp *.* /mnt/mine
16. Edit the device.map
    
    • cd /mnt/mine/grub
    • nano device.map
    • Before:
    • After:
17. Edit the GRUB bootloader to load the shim and the kernel.
    
    1. nano menu.lst
    2. comment the ‘hiddenmenu’ option and increase the timeout so I can test.
       
       • 
    3. Create a new entry specific to the shim and the distribution kernel-xen
       
       • Notice that the kernel is the shim copied from the previous step and the existing kernel and initrd load as modules of the shim.
       • 
    4. Modify the default selection to point to my new entry.
       
       • The default entry begins counting at “0”
       • 
18. Unmount the virtual disk and the cdrom
    
    • cd /
    • umount /dev/hda1
    • umount /dev/hdd
19. Shutdown the virtual machine and remove the ISOs from the DVD drive (also remove the second virtual dvd drive).
20. Boot the virtual machine, note the new menu selection that was created – this is the kernel that should boot.
    
    • 

Note: If you run VMware – This will not run on VMware, the shim is specific to Hyper-V.

Linux vm from VMware to XenServer the videos

If you have been following, you will note quite a few posts related to importing / migrating Linux virtual machines from VMware to XenServer.

I realize that many folks don’t use XenServer – but the basic steps of repairing after migration apply to Hyper-V just as well as to XenServer – the steps are the same if you want your VM to boot. However, PV enablement on Hyper-V does not exist, you just need to install the vm tools.

Here are the links in case you missed them:

• PV enabling an HVM from VMware on XenServer (CentOS RedHat)
• PV enabling an HVM from VMware on XenServer (SLES and Debian)
• Migrating Debian from VMWare ESX to XenServer
• Migrating SLES from VMWare ESX to XenServer
• Migrating CentOS from VMWare ESX to XenServer
• Migrating RedHat from VMware to XenServer

I have turned three of these into short (less than 10 minutes) video presentations, just to add a bit more information than in the articles.

• Repairing SLES 10 boot loader after import to XenServer
• PV enabling SLES 10 vm after import to XenServer
• PV enabling CentOS VM after import to XenServer

PV enabling an HVM from VMware on XenServer (SLES and Debian)

As a condition for paravirtualization to work, a kernel that supports the Xen hypervisor needs to be installed and booted in the virtual machine. Simply installing the XenServer tools within the vm does not enable paravirtualization of the vm.

In this example; the virtual machine was exported as an OVF package from VMware and imported into XenServer using XenConvert 2.0.1.

Installing the XenServer Supported Kernel:

1. After import, boot the virtual machine and open the console.

2. (optional) update the modules within the vm to the latest revision

a. If the kernel-xen package is installed from an online repository – best practice is to fully update the distribution to avoid problems between package build revisions.

3. Install the Linux Xen kernel.

a. yast install kernel-xenpae

i. the xen aware kernel is installed and entries are created in grub

ii. x64 can use kernel-xen, x86 requires kernel-xenpae

iii. This is not the same as installing “xen” which installs a dom0 kernel for running vms, not a domU kernel for running as a vm.

iv. yast is the package installer for SLES, Debian uses apt (apt-get).

4. Modify the grub boot loader menu (the default entries are not pygrub compatible)

Open /boot/grub/menu.lst in the editor of your choice

a. Remove the kernel entry with ‘gz’ in the name

b. Rename the first “module” entry to “kernel”

c. Rename the second “module” entry to “initrd”

i. SuSE and Debian require that entries that point to root device locations described by a direct path such as: “/dev/hd*” or “/dev/sd*” be modified to point to /dev/xvd*

d. (optional) Modify the title of this entry

e. Edit the line “default=” to point to the modified xen kernel entry

i. The entries begin counting at 0 – the first entry in the list is 0, the second entry is 1 and so on

ii. In our example the desired default entry “0”

f. (optional) Comment the “hiddenmenu” line if it is there (this will allow a kernel choice during boot if needed for recovery)

g. Save your changes

1. Edit fstab because of the disk device changes

a. open /etc/fstab in the editor of your choice.

b. Replace the “hd*” entries with “xvd*”

c. Save changes

2. Shut down the guest but do not reboot.

a. Shutdown now -h

Edit the VM record of the SLES VM to convert it to PV boot mode

In this example the VM is named “sles”

5. From the console of the XenServer host execute the following xe commands:

a. xe vm-list name-label=sles params=uuid (retrieve the UUID of the vm)

b. xe vm-param-set uuid=<vm uuid> HVM-boot-policy=”” (clear the HVM boot mode)

c. xe vm-param-set uuid=<vm uuid> PV-bootloader=pygrub (set pygrub as the boot loader)

d. xe vm-param-set uuid=<vm uuid> PV-args="console=tty0 xencons=tty" (set the display arguments)

i. Other possible options are: “console=hvc0 xencons=hvc” or “console=tty0” or “console=hvc0”

6. xe vm-disk-list uuid=<vm uuid> (this is to discover the UUID of the interface of the virtual disk)

7. xe vbd-param-set uuid=<vbd uuid> bootable=true (this sets the disk device as bootable)

The vm should now boot paravirtualized using a Xen aware kernel.

When booting the virtual machine, it should start up in text-mode with the high-speed PV kernel. If the virtual machine fails to boot, the most likely cause is an incorrect grub configuration; run the xe-edit-bootloader (i.e. xe-edit-bootloader –n sles) script at the XenServer host console to edit the grub.conf of the virtual machine until it boots.

Note: If the VM boots and mouse and keyboard control does not work properly, closing and re-opening XenCenter generally resolves this issue. If the issue is still not resolved, try other console settings for PV-args, being sure to reboot the vm and close and re-open XenCenter between each setting change.

Installing the XenServer Tools within the virtual machine:

Install the XenServer tools within the guest:

1. Boot the paravirtualized VM (if not already running) into the xen kernel.

2. Select the console tab of the VM

3. Select and right-click the name of the virtual machine and click "Install XenServer Tools"

4. Acknowledge the warning.

5. At the top of the console window you will notice that the "xs-tools.iso" is attached to the DVD drive. And the Linux device id within the vm.

6. Within the console of the virtual machine:

a. mkdir /media/cdrom (Create a mount point for the ISO)

b. mount /dev/xvdd /media/cdrom (mount the DVD device)

c. cd /media/cdrom/Linux (change to the dvd root / Linux folder)

d. bash install.sh (run the installation script)

e. answer “y” to accept the changes

f. cd ~ (to return to home)

g. umount /dev/xvdd (to cleanly dismount the ISO)

h. In the DVD Drive, set the selection to “<empty>”

i. reboot (to complete the tool installation)

7. Following reboot the general tab of the virtual machine should report the Virtualization state of the virtual machine as “Optimized”

Distribution Notes

Many Linux distributions have differences that affect the process above. In general the process is similar between the distributions.

Removal of VMware Tools was tested following import to XenServer and I do not recommend removal of VMware Tools after the VM has been migrated to XenServer. If it is desired to remove VMware Tools, the vm must be running on a VMware platform when the uninstall command is executed within the VM ( rpm -e VMwareTools ).

Some distributions have a kernel-xenpae in addition to the kernel-xen. If PAE support is desired (or required) in the virtual machine, please substitute kernel-xenpae in place of kernel-xen in the instructions. Please see the distribution notes for full details.

PV enabling an HVM from VMware on XenServer (CentOS RedHat)

This example works for RedHat and CentOS, the instructions are slightly different for SLES and Debian.
As a condition for paravirtualization to work, a kernel that supports the Xen hypervisor needs to be installed and booted in the virtual machine.
Installing the XenServer Supported Kernel:
1. After importing the vm as HVM, boot the virtual machine and open the console.
2. (optional) update the modules within the vm to the latest revision
a. If the kernel-xen package is installed from an online repository – best practice is to fully update the distribution to avoid problems between package build revisions.
3. Install the Linux Xen kernel.

a. yum install kernel-xen

i. the xen aware kernel is installed and entries are created in grub

4. Build a new initrd without the SCSI drivers and with the xen PV drivers

a. cd /boot

b. mkinitrd --omit-scsi-modules --with=xennet --with=xenblk --preload=xenblk initrd-$(uname -r)xen-no-scsi.img $(uname -r)xen

i. This builds a new initrd for booting with pygrub that does not include SCSI drivers which are known to cause issues with pygrub and Xen virtual disk devices.

5. Modify the grub boot loader menu (the default entries are not pygrub compatible)
Open /boot/grub/menu.lst in the editor of your choice

a. Remove the kernel entry with ‘gz’ in the name

b. Rename the first “module” entry to “kernel”

c. Rename the second “module” entry to “initrd”

i. SuSE and Debian require that entries that point to root device locations described by a direct path such as: “/dev/hd*” or “/dev/sd*” be modified to point to /dev/xvd*

d. Correct the *.img pointer to the new initrd*.img created in step 4

e. (optional) Modify the title of this entry

f. Edit the line “default=” to point to the modified xen kernel entry

i. The entries begin counting at 0 – the first entry in the list is 0, the second entry is 1 and so on

ii. In our example the desired default entry “0”

g. (optional) Comment the “hiddenmenu” line if it is there (this will allow a kernel choice during boot if needed for recovery)

h. Save your changes

6. Shut down the guest but do not reboot.

a. Shutdown now -h

Edit the VM record of the CentOS VM to convert it to PV boot mode
In this example the VM is named “centos”
7. From the console of the XenServer host execute the following xe commands:

a. xe vm-list name-label=centos params=uuid (retrieve the UUID of the vm)

b. xe vm-param-set uuid=<vm uuid> HVM-boot-policy=”” (clear the HVM boot mode)

c. xe vm-param-set uuid=<vm uuid> PV-bootloader=pygrub (set pygrub as the boot loader)

d. xe vm-param-set uuid=<vm uuid> PV-args="console=tty0 xencons=tty" (set the display arguments)

i. Other possible options are: “console=hvc0 xencons=hvc” or “console=tty0” or “console=hvc0”

8. xe vm-disk-list uuid=<vm uuid> (this is to discover the UUID of the interface of the virtual disk)
9. xe vbd-param-set uuid=<vbd uuid> bootable=true (this sets the disk device as bootable)

The vm should now boot paravirtualized using a Xen aware kernel.
When booting the virtual machine, it should start up in text-mode with the high-speed PV kernel. If the virtual machine fails to boot, the most likely cause is an incorrect grub configuration; run the xe-edit-bootloader (i.e. xe-edit-bootloader –n centos) script at the XenServer host console to edit the grub.conf of the virtual machine until it boots.
Note: If the VM boots and mouse and keyboard control does not work properly, closing and re-opening XenCenter generally resolves this issue. If the issue is still not resolved, try other console settings for PV-args, being sure to reboot the vm and close and re-open XenCenter between each setting change.
Installing the XenServer Tools within the virtual machine:
Install the XenServer tools within the guest:
1. Boot the paravirtualized VM (if not already running) into the xen kernel.
2. Select the console tab of the VM
3. Select and right-click the name of the virtual machine and click "Install XenServer Tools"
4. Acknowledge the warning.
5. At the top of the console window you will notice that the "xs-tools.iso" is attached to the DVD drive. And the Linux device id within the vm.
6. Within the console of the virtual machine:

a. mkdir /media/cdrom (Create a mount point for the ISO)

b. mount /dev/xvdd /media/cdrom (mount the DVD device)

c. cd /media/cdrom/Linux (change to the dvd root / Linux folder)

d. bash install.sh (run the installation script)

e. answer “y” to accept the changes

f. cd ~ (to return to home)

g. umount /dev/xvdd (to cleanly dismount the ISO)

h. In the DVD Drive, set the selection to “<empty>”

i. reboot (to complete the tool installation)

7. Following reboot the general tab of the virtual machine should report the Virtualization state of the virtual machine as “Optimized”

XenConvert 2.0.1 is released with VMware OVF compatibility

We have been working on adding Citrix Project Kensho OVF capabilities to XenConvert.

XenConvert is the free Citrix machine conversion utility.  It is primarily focused on converting workloads to either Provisioning Server or to XenServer, however there are some more generic functions that are of interest to most any virtualization folk.

The download can be found here:

http://www.citrix.com/English/ss/downloads/details.asp?downloadId=1855017&productId=683148

If this moves in the future go here: http://www.citrix.com/English/ss/downloads/results.asp?productID=683148 and look for XenConvert in the XenServer download section.

OVF packages from any existing VMware product (known to this date) can be consumed (imported) direct to XenServer.

The physical to OVF path can be run within a Windows machine and convert it to an OVF (meta file + .vhd) or just a vhd.

The OVF can then be consumed to XenServer with XenConvert 2 or to XenServer and / or Hyper-V in the upcoming Kensho refresh.

The VHD can, of course, be copied to any engine that uses vhd.

It also does a binary conversion of vmdk to vhd and injects a critical boot device driver that is compatible with XenServer (and works with Hyper-V).

Also, the XenServer .xva (vm backup files) can be converted to OVF.

Download and enjoy!

Migrating Debian from VMWare ESX to XenServer

Fixing Debian virtual machines after migration
After import:  Debian boots, detects new hardware, and installs new device drivers.
The boot sequence fails at the following error: ALERT! /dev/sda1 does not exist. Dropping to a shell!

This is because the source host (ESX server) presents boot devices as SCSI bus devices by default and the paths have been set within the installation of Debian.
To fix the Debian boot loader:
The Grub boot loader has dropped you into the BusyBox recovery shell. At a command prompt under initramfs.
Begin by looking at the Storage tab in XenCenter and note the presentation (device path) of the virtual disk. This describes the bus that is used to present the virtual disk to the virtual machine. This information does not appear for all virtual machines, but in this case it is available so I will use it to save time attempting to discover the interface and volumes.

This shows that the virtual disk is on /dev/hda – this represents the first IDE interface.
I will begin by returning to the virtual machine console and listing the IDE devices.

Five IDE devices have been identified.
· /hda represents the disk in position 0 IDE controller 0
· /hda1 represents the first partition of the disk /hda
I am going to make an assumption that my Debian installation is on the first partition of the IDE interface. Please note that some Linux installations place the swap on first volume, so some repetition of the follow steps might be necessary to discover the proper volume.
At the command prompt I am going to mount the partition as /root.
This is safe in this case because /root was not loaded as noted by the Alert! message identifying the missing boot device.

Now that the file system is mounted under /root I will verify that it appears to be my Linux installation

Listing the contents I see \boot, \root, \etc, \bin, and so forth. All of these are directories that I expect to find at the root of a Linux installation.
I will first fix the device descriptor file /etc/fstab.
· Change to the /root/etc directory
· Open fstab in an editor

Note the entries that point to /dev/sda; these are the entries that we will be changing to /dev/hda since a SCSI disk controller is no longer available.
Modify the /sda entries and save the file.

Similar steps need to be repeated for the Grub boot loader and the device map.
Change to the /grub folder.

Open the menu.lst file in an editor.
*Note: This file name is menu.Lst not menu.1st
Near the end of the file are the entries that we are concerned with.
Find the boot menu options that point to the /sda device.

Change the entries to /hda. Then save the file.

The virtual machine is now ready for reboot.
Simply enter ‘reboot’ at the command prompt.
During reboot, you may notice that the boot hangs after detecting the USB Tablet device. Select <Enter> to accept the default identified mouse device.
Previously under VMware, my Debian virtual machine was running X server for a graphical desktop.
As with RedHat an error that X server failed to start. After selecting No to bypass the detailed troubleshooting screens a dialog is presented that the X server has been disabled.

After selecting OK you are presented with a logon prompt.
To repair the X server:
To reconfigure the X server two methods can be used:
1) Edit the file /etc/X11/xorg.conf directly
2) Allow automated reconfiguration and accept the identified devices.
To run the automated reconfiguration:
Login as root and then execute reconfiguration for the xserver package.

Step through the wizard and accept the automatically detected defaults.
When the wizard completes start xserver by entering startx or reboot the virtual machine.

Migrating SLES from VMWare ESX to XenServer

For SuSE Linux Enterprise Server I require a “Helper” virtual machine to mount and repair the file system. This is because SLES recovery console does not include an editor.

After migrating SuSE and booting the boot loader fails at: “waiting for device /dev/sda2.” This is as expected because /sda refers to a SCSI bus and on XenServer SuSE actually sees an /hda (IDE) boot device.

The Helper VM can be created using the XenServer provided Debian Etch template virtual machine (this template includes the media, making it practically ready to go). The included Debian distribution also works with the SUSE reiserFS that is installed by default.

It is also of note that SuSE has a full Xen aware kernel and can be further optimized by presentation of the boot devices as Xen Virtual Disks and by loading a paravirtualized kernel. These optimizations are outside of this article; this is specifically focused on having a running virtual machine.

Import the VM to XenServer:

In my examples I am using XenConvert 2.0 to consume the VMware OVF virtual appliances, however Citrix Project Kensho can also be used.

Creating the Helper virtual machine:

In XenCenter select VM -> New

Choose ‘Debian Etch 4.0’ as the template (this template provides the installation template plus the operating system, nothing to download).

Name the virtual machine “HelperVM” and complete the New VM wizard accepting the defaults, allow the VM to boot, and open the console of this VM.

At the console of HelperVM enter a new root password, VNC password, and a host name (‘HelperVM’ is my suggestion).

Mount the SLES virtual disk to HelperVM:

In XenCenter select the SuSE virtual machine, and then select the Storage tab.

Select the virtual disk (make a note of the disk name) and then click Detach.

*Note: the VM must be powered off to detach a virtual disk.

Select HelperVM, then the Storage tab, and then click Attach.

Select the SuSE virtual disk from the Storage Repository and click Attach.

Return to the HelperVM console.

Note that HelperVM should have auto-mounted the volume (in this example HelperVM was running when I attached the virtual disk). My example added the controller device xvdc with the partitions of xvdc1 and xvdc2.

This can also be seen in the Storage tab of XenCenter.

Return to the console of HelperVM and create a path to mount the volume and mount the first volume.

Mkdir /mnt/suse

mount /xvdc/xvdc1 /mnt/suse

Note the error that this looks like swap. I will try to mount the other volume.

Switch to the mounted file system and list to verify that this appears to be the root volume.

Repairing the boot loader:

From this point forward the process is fundamentally no different than repairing Debian or modifying the Grub menu and fstab of any Linux distribution.

I will begin by repairing fstab.

From the root of the mounted SuSE virtual hard disk (/mnt/suse) change to the /etc directory and open the fstab file in an editor.

It should look similar to my nano editor screen below:

I am going to modify the two entries that point specifically to a SCSI presented boot device to IDE.

Previously, in the XenCenter Storage tab for the SuSE virtual machine I observed that the virtual disk was presented on an IDE controller.

The new fstab should resemble this:

Now, to proceed to the Grub boot loader menu.

One way to approach this is to copy an existing entry to a new entry and make the necessary modifications to the new entry. In this example I am modifying the existing entries for the new hypervisor.

Change to the /boot/grub directory ( cd /mnt/suse/boot/grub )

And open menu.lst in an editor.

Find the entries that refer to /dev/sdaX and change them to /dev/hdaX. In the screen shot above this is /dev/sda2

Then save the modifications.

To safely continue I need to un-mount the SuSE virtual disk from the HelperVM.

Return to the root of the file system ( cd / ) and use the umount command to un-mount the xvdc2 device.

To continue to repair the SuSE virtual machine, the virtual disk needs to be detached from the HelperVM and attached back to the SuSE virtual machine.

Mount the SLES virtual disk to the SuSE VM:

Begin by shutting down HelperVM.

Select the Storage tab of HelperVM. Select the SuSE virtual disk and select Detach.

Select the SuSE virtual machine, select the Storage tab, click Attach

Select the correct virtual disk and Attach.

Open the console of the SuSE virtual machine and power it on.

Additional repairs:

As with other Linux distributions, if X server was used to present a graphical console it will require repair due to the capabilities of new video devices.

X server is then disabled.

To repair X server, logon as root and repair the X server by running SaX2

At the command prompt execute the command sax2 -f

At the completion of the wizard X server can be started by executing startx or rebooting.

The one thing that you will notice is that there is no mouse support. Setting up VNC Server within the virtual machine and connecting to a graphical console using this VNC connection can resolve this situation.