Saturday, June 27, 2009

PV enabling an HVM from VMware on XenServer (CentOS RedHat)

This example works for RedHat and CentOS, the instructions are slightly different for SLES and Debian.
As a condition for paravirtualization to work, a kernel that supports the Xen hypervisor needs to be installed and booted in the virtual machine.
Installing the XenServer Supported Kernel:
1. After importing the vm as HVM, boot the virtual machine and open the console.
2. (optional) update the modules within the vm to the latest revision
a. If the kernel-xen package is installed from an online repository – best practice is to fully update the distribution to avoid problems between package build revisions.
3. Install the Linux Xen kernel.
a. yum install kernel-xen
i. the xen aware kernel is installed and entries are created in grub
4. Build a new initrd without the SCSI drivers and with the xen PV drivers
a. cd /boot
b. mkinitrd --omit-scsi-modules --with=xennet --with=xenblk --preload=xenblk initrd-$(uname -r)xen-no-scsi.img $(uname -r)xen
i. This builds a new initrd for booting with pygrub that does not include SCSI drivers which are known to cause issues with pygrub and Xen virtual disk devices.
5. Modify the grub boot loader menu (the default entries are not pygrub compatible)
Open /boot/grub/menu.lst in the editor of your choice
a. Remove the kernel entry with ‘gz’ in the name
b. Rename the first “module” entry to “kernel”
c. Rename the second “module” entry to “initrd”
i. SuSE and Debian require that entries that point to root device locations described by a direct path such as: “/dev/hd*” or “/dev/sd*” be modified to point to /dev/xvd*
d. Correct the *.img pointer to the new initrd*.img created in step 4
e. (optional) Modify the title of this entry
f. Edit the line “default=” to point to the modified xen kernel entry
i. The entries begin counting at 0 – the first entry in the list is 0, the second entry is 1 and so on
ii. In our example the desired default entry “0”
g. (optional) Comment the “hiddenmenu” line if it is there (this will allow a kernel choice during boot if needed for recovery)
h. Save your changes
6. Shut down the guest but do not reboot.
a. Shutdown now -h
Edit the VM record of the CentOS VM to convert it to PV boot mode
In this example the VM is named “centos”
7. From the console of the XenServer host execute the following xe commands:
a. xe vm-list name-label=centos params=uuid (retrieve the UUID of the vm)
b. xe vm-param-set uuid=<vm uuid> HVM-boot-policy=”” (clear the HVM boot mode)
c. xe vm-param-set uuid=<vm uuid> PV-bootloader=pygrub (set pygrub as the boot loader)
d. xe vm-param-set uuid=<vm uuid> PV-args="console=tty0 xencons=tty" (set the display arguments)
i. Other possible options are: “console=hvc0 xencons=hvc” or “console=tty0” or “console=hvc0”
8. xe vm-disk-list uuid=<vm uuid> (this is to discover the UUID of the interface of the virtual disk)
9. xe vbd-param-set uuid=<vbd uuid> bootable=true (this sets the disk device as bootable)
The vm should now boot paravirtualized using a Xen aware kernel.
When booting the virtual machine, it should start up in text-mode with the high-speed PV kernel. If the virtual machine fails to boot, the most likely cause is an incorrect grub configuration; run the xe-edit-bootloader (i.e. xe-edit-bootloader –n centos) script at the XenServer host console to edit the grub.conf of the virtual machine until it boots.
Note: If the VM boots and mouse and keyboard control does not work properly, closing and re-opening XenCenter generally resolves this issue. If the issue is still not resolved, try other console settings for PV-args, being sure to reboot the vm and close and re-open XenCenter between each setting change.
Installing the XenServer Tools within the virtual machine:
Install the XenServer tools within the guest:
1. Boot the paravirtualized VM (if not already running) into the xen kernel.
2. Select the console tab of the VM
3. Select and right-click the name of the virtual machine and click "Install XenServer Tools"
4. Acknowledge the warning.
5. At the top of the console window you will notice that the "xs-tools.iso" is attached to the DVD drive. And the Linux device id within the vm.
6. Within the console of the virtual machine:
a. mkdir /media/cdrom (Create a mount point for the ISO)
b. mount /dev/xvdd /media/cdrom (mount the DVD device)
c. cd /media/cdrom/Linux (change to the dvd root / Linux folder)
d. bash (run the installation script)
e. answer “y” to accept the changes
f. cd ~ (to return to home)
g. umount /dev/xvdd (to cleanly dismount the ISO)
h. In the DVD Drive, set the selection to “<empty>”
i. reboot (to complete the tool installation)
7. Following reboot the general tab of the virtual machine should report the Virtualization state of the virtual machine as “Optimized”


Anonymous said...

Hi great tutorial.
Testet with a VM from VMWare2.0 to XenServer 5.0.
I Have only Problems with the PV-args. I can see everything on the xen center but can not edit something into the console.
with PV-args="console=tty0 xencons=tty" the console hangs.
With PV-args="" or PV-args="console=hvc0 xencons=hvc" i can see it but have no edit function.

BrianEh said...

There are many PV-args options that are possible.

What do you mean by: "I can see it but have no edit function."

This should be editable through xe commands at any time that the VM is not running.

Sometimes only the console= is required.
Sometimes you might also see a console= declaration in the bootloader entry. And I have better luck when I delete it from the grub entry.

I have found that I generally have to play around with it.

Oh! And if you change this console setting and you are using XenCenter - you will most likely have to close and re-open XenCenter. The only reason that I can figure for this is that the video is captured following editing this setting, but the mouse and keyboard are lost.

Jeremy said...

Just wanted to say this post solved an issue unrelated to what it was written about but none-the-less saved me hours of work. Thanks!

daniel said...

Hi, great infos! they saved me hours of search.

I followed the tutorial applying it to a Fedora 12, the shipped kernel already supports booting as DomU and stock grub.conf looks already pygrub compatible.

but pygrub fails to find the partition with the kernel on it! what might it be?
I just realized that the VM boots in a LVM which may not be supported by xenserver, any ideas?

BrianEh said...

I have not tried this with Fedora 12, so I am only specilating here.

I have generally found that the RedHat and Fedora installation - 'just work' when moving them about as an HVM type VM.

The only way that this will work as a PV'd machine is if Fedora has a kernel-xen and then when the kern-xen is loaded new disk devices come into play - thi sis where the xvd* type devices socome from.

LVM should not matter. But most likely the device driver that we expect is not loading properly and the device that Grub is looking for is not there.

I have also seen QEMU devices appear as well.
In general, Anaconda picks these hardware changes up.

I know that I have not directly answered this, but I am not familiar wit hte xen PV kernel in Fedora 12 enough to provide any concrete answers.

Jorge said...

Hi, can you post a guide for Paravirtualizing a Linux VM not coming from VMware but installing from scratch - ISO/DVD?

In my case I need to install ClearOS Gateway ( based in CentOS 5.4, I can't install it using CentOS 5.3 template in Xenserver so I will need some of your help.

I also posted this problem in Citrix Forums:

Thank you very much!

BrianEh said...

The steps are nearly the same.

Optimally, you want to use the XenServer provided template - it is more than just VM hardware settings, far more.

In your case, you create a new VM as an HVM (just as if it came from VMware) with one of the Windows templates. (do not use 'other').

Then you install the OS.
Add the kernel-xen and shut down the vm - then in the XenServer cli you modify the VM to make it PV.

Then boot the VM and in the boot menu select the kernel-xen. You now have a VM that can boot as either a paravirtualized or HVM type machine. Making it uber portable.

Anil outlined the steps for Ubuntu here:

Jorge said...

I actually tried to use (and I prefer it) the Xenserver provided template for CentOS 5.3 but I'm always asked for a CDROM driver and can't get away with it that's why I can't continue. Also even NFS and FTP installation doesn't work. Maybe it's because of the 5.3 difference over 5.4? You have any suggestion?

Trying your guide now on a Windows 2003 template. Will let you know the results.

Jorge said...

I was able to successfully install ClearOS 5.1 (based in CentOS 5.4) xen kernel and follow your instructions without problem. I did get my VM paravirtualized!

But after looking to all possible causes I still can't make the NIC's in my VM to work and connect. The NIC are present and detected by the VM but can't get an IP which is working well before doing the PV guide.

Result of ifconfig -a && cat /etc/sysconfig/network-scripts/ifcfg-eth0:

Content of /var/xapi/state.db:

Please help me make it work, been working this for 2 days already :(

Thanks a lot!

Anonymous said...

Thank you!
This worked perfectly in order to get my CentOS 5.4 installation to work with XenServer 5.5.

I had to set the PV-args to an empty string (""), and when I booted the machine, the network configurations settings file was renamed making the network disabled.

Just renamed the file right back, and everything works!

Peter Kline said...

Great article! - one catch I had of note - might even want to be part of the article.

On my CentOS 5.5 distro, I *HAD* to rebuild the init ramdisk. I was misled into not doing this when `yum install` placed an image with the same name in the /boot directory. Nonetheless, it wouldn't find the block devices and thus my LVM PVs until I rebuild the initrd with the xenblk drivers.

Also - if you are accessing the console via XenCenter you must restart the XenCenter software or it can cause issues with your keyboard not making it to the console.

Sax Appeal said...

This post is really a golden nugget.
Thanks a 1000.

Anonymous said...

Hi, I run into same problem with NICs as Jorge.
My xenservr is 5.6.0, host os is ClearOS 5.2.

BrianEh said...

These new articles might be useful to you:

Installing Ubuntu Server 10.04 (32bit and 64bit) LTS

Configuring RHEL 5.5 Guests on XenServer 5.5 and 5.6

Configuring CentOS 5.5 Guests on XenServer 5.5 and 5.6

Unknown said...

Thanks Brian for this, if I had found it sooner it would have saved me a bunch of hours of banging my head against a wall trying to PV-ise a VM which had been setup incorrectly.

FYI I had to go with the PV-args="console=hvc0" also. My next challenge is to get the graphical UI working again, but I'll tackle that another day :)



SDB69 said...

After a week of trying to get XenServer Tools installed, then realizing the converted CentOS 5.5 64-bit was a HVM, but needed to be a PV... your blog was hands down the most useful information that I found on the topic. Thank you much for the help.

BrianEh said...

This post is getting a bit old. But folks contribute over time to the things I can't.

Here is some new help that came through a Citrix XenConvert forum:

I needed to update my kernel for the mkinitrd step to work. Otherwise I'd end up with the error "no modules available for kernel". To update your kernel, run "yum -y update kernel kernel-headers". Be sure to reboot once you've done this.

The initial PV-args setting that the article suggests caused my console to not allow keyboard input. The PV-args setting that ultimately worked for me was “console=tty0”.

The cdrom device that you need to mount for the xen-tools.iso installation step will vary. For us, it was /dev/xvdc instead of /dev/xvdd as the article suggests.

Unknown said...

Followed these instructions for Centos 5.8 based distro, & it worked like a charm!!
full PV mode is working great!! thank you

Marllus Lustosa said...


Marllus Lustosa said...


Aaron Nichols said...

Brian, this post is now my go to for HVM to PV conversions. I feel like I owe 2009 you a case of beer for how handy this is. The only other thing that has helped as much as this (which, I did notice a lot of exists already in the official documentation, but your narration is gold) is the ./pvtools script out on the web.

I figure today would be about the 50th time I've opened this post, so I decided to finally drop a thank you.

Anonymous said...

Brian, this post is now my go to for HVM to PV conversions. I feel like I owe 2009 you a case of beer for how handy this is. The only other thing that has helped as much as this (which, I did notice a lot of exists already in the official documentation, but your narration is gold) is the ./pvtools script out on the web.

I figure today would be about the 50th time I've opened this post, so I decided to finally drop a thank you.