Showing posts with label OVF. Show all posts
Showing posts with label OVF. Show all posts

Friday, August 30, 2013

Zip files and folders with PowerShell

One of the more frustrating things in this day of PowerShell v3 is not having cmdlets that can simply manipulate ZIP archives.

It is right there in the Windows GUI, but is it easy to automate?  Nope.

In fact if you search around you will find lots of different ways to handle this, one getting more complex than the next.  You will also find community projects that attempt to do the same thing.

One common reference that I ran across was this:

http://blogs.msdn.com/b/daiken/archive/2007/02/12/compress-files-with-windows-powershell-then-package-a-windows-vista-sidebar-gadget.aspx

From the spectacular David Aiken. I have to admit, it is not the first time he has saved my bacon.

His solution is built in, no funky community add-ins, nothing strange I can’t follow, and best of all it is compact – it is really small.

I have already had to do some things with Shell.Application with PowerShell, so I figured I would give it a shot.

Well, I immediately ran into some issues. 

One, his use of –Recurse.  Not necessary.  Use Get-ChildItem and pipe in the folder and then entire folder is zipped.  Perfect.  So you can work the input just about any way you like and it will simply pass whatever you pipe to it.

Two, file locking.  This tripped my up for hours.  And lots of other folks that have found his solution too.  His little 500 millisecond wait before advancing on to the next is simply not real nor flexible based on varying file sizes.

I found all kinds of folks commenting on the same thing and developing all kinds of fancy solutions to handle it.  But, in the end I found something really simple, and it was buried right there in the shell.application all along.  Simply test for the existence of the item you are zipping in the zip archive.

So simple.  One little do loop.  With my crafty Until ( $zipPackage.Items() | select {$_.Name -eq $file.Name} )

And, I only modified the Add-Zip, since David already had a fail safe to create the .ZIP if it didn’t already exist.

I am going to update my use of his usage example as well.

In my case I have a number of XML files.  Each is in a unique Folder.  There are other files and folders with the XML files as well (this is an OVF, for you OVF fans).

I want to create the .ZIP one level up from the folder where the XML is.

$ovfFolder = Get-Item $xmlFile.PSParentPath 

$zipFullName = $ovfFolder.Parent.FullName + "\" + $xmlFile.BaseName + ".zip"

Get-ChildItem $ovfFolder | Add-Zip $zipFullName

Now, below is my modification to the original Add-Zip function.

function Add-Zip  # usage: Get-ChildItem $folder | Add-Zip $zipFullName
{
    param([string]$zipfilename)

    if(!(test-path($zipfilename)))
    {
        set-content $zipfilename ("PK" + [char]5 + [char]6 + ("$([char]0)" * 18))
        (dir $zipfilename).IsReadOnly = $false   
    }
    $shellApplication = new-object -com shell.application
    $zipPackage = $shellApplication.NameSpace($zipfilename)
    foreach($file in $input)
    {
        $zipPackage.CopyHere($file.FullName)
        do {
            Start-sleep 2
        } until ( $zipPackage.Items() | select {$_.Name -eq $file.Name} )
    }
}

Friday, May 7, 2010

OVF vs OVA the saga continues

It has been over two years since I began working with the OVF standard from the DMTF.

Repeatedly during this process I have had to educate folks about what an OVF package is, and quite frequently what an OVA is and when to use it.  Just yesterday I corrected a person in a conference call as this is still a relatively new thing to most folks.

In very simple terms, an OVA is a single file.  It is a TAR archive of an OVF.  It simply has the file extension “.ova” to give some indication that it is an OVF.  Otherwise it would be just any other “.tar”

The OVF is the real important part that the DMTF keeps defining and expanding upon.

The OVF package is basically two things. 

  1. It is an XML file that describes the virtual environment (vCPU, vRAM, vNetwork config, and it lists references to any other parts of the OVF package).
  2. It is a collection of files.  Virtual disks of virtual machines, .ISO installation media, any other file attachments that a person could dream up.

Use OVA when you need to take an OVF package somewhere.  Or want to give it to someone as a download.

Use an OVF for all of your internal purposes.

Is OVF a promise that an appliance can be imported to and run on any hypervisor?  Absolutely not.

OVF is NOT a way to convert workload from one platform to another – no reason why it could not be used that way (it makes sense) – but that is not why it exists.  Eventually I see companies implement conversions around OVF packages.  The closest thing I know of today is the “fixup” that is in the Citrix Kensho implementation.

There are two reasons why:  The OS installed in the VM and its built in driver support and that different hypervisors present virtual hardware in different ways. 

To a lesser extent there is a third reason: proprietary VM tools.  The tools from vendor A can prevent a VM from running on a different hypervisor (or at least make it really difficult) or they can cause the performance of the VM to be very poor on the new hypervisor.

So, the moral of the story:  Use OVA appropriately and don’t expect an OVF package build on one hypervisor to “just work” on a different hypervisor.  And be wary of VM tools installed within the VM.

Friday, January 1, 2010

IT needs a language

I constantly struggle with writing documentation and also communicating concepts to many IT folks in writing. 

This is one of the reasons that has driven me to producing videos and capture sessions over the past couple years (I can say far more, more accurately with combined speech and visuals than attempt to accurately convey in writing).

My work as a moderator of the Hyper-V TechNet forum also speaks to this.  I spend a great deal of time describing concepts in forum posts – that end up being incredibly long.  Each time I write it a bit differently, still trying to convey a concept.

The reason for this post?  Accuracy.  Accurate conveyance of information.

This all started with a recent trip to the doctor.  20 minutes of discussing things with my doctor was summarized into a single paragraph of Latin.  The use of the common language of Latin to describe symptoms, and convey that description from doctor to doctor.

I am not proposing that Latin be revived in IT.  However, to use commonly understood grammatical structure to describe concepts, or installations, or topology, or systems architecture.

This gets even more complex as we move into the virtual workload world.  Where we describe physical characteristics in a non-physical way.

A virtual appliance for example.  In its most common representation today, this is a virtual machine – installed within that virtual machine is an operating system and some application.  The application and OS have dependencies – DNS, networking, Active Directory authentication for example.  The machine has dependencies – two virtual NICs, one public facing, one that faces a private subnet that connects to the database server.  Oh, and now we have an external entity of a database server.  Where does it all end – I don’t know of a single workload that exists in an enterprise as a single entity.

The struggle is:  How do we describe this in a universal way so that no matter where you take that appliance the environment knows what to do with it?  How to configure it.  How to connect it.  Where to connect it.

This is the role of a Standards Body – in this case the DMTF is attempting to build this common term framework.  But this is a framework, it is not universal terms – it is simply items or entities.  So at least we begin with common items – however describing them in different languages by vendor.

Now, how do I take that DMTF developed structure and tern it into words.  One set of words that can be used in association with a NIC to describe its network connection and then the attachment and VPN attributes.

Or, throw that idea out the window and focus on describing the interactions between the workloads only.  Don’t describe the physical topology in any way – simply describe the dependencies between workloads or appliance entities.

Most IT folks would never see this description language – but integrators would find it useful, documentation folks would find it useful, and even developers would find it useful.

Thursday, December 17, 2009

Datamation puts their angle on the big three virtualization vendors

Datamation has put their spin and opinion into the world of virtualization and the virtualization engines and offerings in this comparison of Citrix (XenServer), Microsoft (Hyper-V, SCVMM), and VMware (ESX, vCenter).

Mind you, this is a judgment article – But it mentions Kensho – a project very near to my personal efforts, and DocFinder – another product I was involved in the infancy of.

You can find the article here:

http://itmanagement.earthweb.com/features/article.php/12297_3853716_1/Virtual-Server-Comparison-Xen-vs-Microsoft-vs-VMware-2010.htm

Friday, October 9, 2009

Importing Kensho OVF to ESX

My main focus has been in taking OVF content from other vendors and consuming that with Citrix Kensho (the OVF Tool or XenConvert 2.x) – however lets turn the tables.
OVF is a format that describes virtual appliances; these could be single or multiple machine.  In doing so, OVF describes the virtual hardware and physical requirements of each machine.
In the spirit of supporting the development of interoperability between vendors I also consider how other vendors consume OVF appliances.
I have my personal opinion of how things should work, but the DMTF VMAN is a committee, therefore there are many opinions.
At the same time, the promise of interoperability is as much in the hand of the consume of an appliance as it is in the creator, as it is in the operating system of the machine in the appliance.
Lets look at using VMware Products (as released today 10/9/2009) to consume a Citrix Kensho created OVF appliance. 
Well, I am going to be honest – at this time VMware does not translate other vendors hardware descriptions into VMware equivalents – therefore VMware cannot import OVF content that does not adhere to the VMware way of describing the hardware of a virtual machine.
So, knowing that, is there a workaround?  Yes, there is.
In my example I am going to use XenConvert to create an OVF Appliance from a XenServer XVA (that is an export of a XenServer VM), modify that using a VMware created OVF, and then import to ESX.
Mind you, this is not for the easily sickened, nor is it something you want to do every day.
First of all, I begin by downloading the Citrix Merchandising Server virtual appliance. I also use XML Notepad, ESX, and VMware Converter.  (Please note that following this process will NOT magically make the Merchandising Server work on ESX - you need to use the appliance built for VMware)
[1] Expand the Merchandizing bz2 zip archive. (WinRAR can do this, and others as well)
[2] Use Citrix XenConvert 2.x to convert from “Xen Virtual Appliance” to “Open Virtualization Format (OVF) Package
Do not create an OVA, just an OVF.
[3] Using the VMware Client:
a. Create a shell VM using Linux / RedHat Enterprise Linux 5 (32-bit)
b. Select the VM that was created and export it to an OVF appliance
· VI3 = Select VM, File, Virtual Appliance, Export
· vSphere4 = Select VM, Export
[4] Open both folders containing the OVF appliance from XenConvert and the OVF appliance from VMware.
Note that both folders contain common items: a virtual disk file, and a .ovf metafile that describes the settings of the appliance.
clip_image002
[5] Copy the VHD from the XenConvert created appliance to the folder of the VMware created appliance
clip_image004
[6] Open the .ovf XML metafile from both appliances using XML Notepad
[7] Open the two references to the virtual disk
There are two important sections that need to be modified in the VMware OVF so that it will properly ‘see’ the VHD. The ‘name’ of the virtual disk and the ‘format’.
clip_image005
[8] Copy the disk reference sections from the XenConvert created appliance to the VMware created appliance.
clip_image006
[9] Save the changes
[10]If the VMware appliance folder contains a file ending in .mf – delete it
[11]Using VMware Converter import the modified VMware OVF appliance.
Only VMware Converter can both consume the OVF and convert the VHD to VMDK.
a. Open VMware Converter
b. Select Convert Machine
c. Choose virtual appliance, browse to the modified OVF
d. Select Next
e. Complete the import wizard
The preceding steps converted the Merchandising Server XVA based appliance to OVF and then used a VMware derived OVF to import the appliance to VMware. The operating system within the appliance still needs to be repaired to allow the machine to boot and run.
Hopefully, I will have a solution for that in the near future.  As there are currently many challenges inherent within the operating system of the machines themselves that prevent true interoperability.

Thursday, September 24, 2009

Enabling Citrix Merchandising Server paravirtualized vm to run on Hyper-V

Be warned, Citrix Support will not support your Merchandising Server if you follow these steps.

And - this is really antique and no longer works with Hyper-V 2012.

Here is my scenario:

I have a fully paravirtualized Linux virtual machine (Merchandising Server) that is made to run on a xen hypervisor (the family of hypervisors that enable Linux VMs to boot kernel-xen and run in a truly paravirtualized and highly efficient way).
I want to move that vm over to another hypervisor (non-xen) for a client demonstration.
Warning:  This might seem a bit convoluted, but it really isn’t difficult.  However there are a few tools involved.  The steps that apply to your situation all depends on how easy it is to get your virtual disk into a format that you can mount and edit the boot volume.  Follow these instructions at your own risk, a positive outcome is not guaranteed.  The resulting VM will most likely not be supported if it has problems.
Collecting the elements:
I am going to use a few things that I need to obtain (download) and tools that need to be installed.

Why so many tools?

The Merchandising Server is the appliance that I am using for the example.  XenConvert is to convert the XVA (XenServer export format) into an OVF based appliance.  The Kensho OVF Tool is to import the OVF into Hyper-V.  The Linux distribution “Live CD” is to mount the virtual disk of the example appliance so we can modify the Grub boot loader and drop a file.  And the v1 Linux Integration Components have the magic PV kernel shim that we need.

Details, details, missing details.  What gives?

I am going to state right now that; I am not going to go into deep and gory details describing each and every click that is required with each tool.  If you read my blog, I assume a couple things: that you have a clue, or you want one.  And that you are not afraid of figuring things out, or trying and failing (I am also implying that you know how to make back-ups of things before you go mucking them up).

The business at hand:

  1. Expand the Merchandizing bz2 zip archive. (WinRAR can do this, and others as well)
  2. Use Citrix XenConvert 2.x to convert from “Xen Virtual Appliance” to “Open Virtualization Format (OVF) Package
    • Do not create an OVA, just an OVF.
  3. Use Citrix Kensho OVF Tool to Import the OVF Package from step 2 to a Hyper-V host.
    • Or you could just copy the VHD from step 2 to your Hyper-V host and create a new VM.
    • Do not boot the VM at this time.
  4. Attach the Live CD ISO to the VM
  5. Set the boot order to boot from DVD first
  6. Remove the default Network Adapter and add a Legacy Network Adapter
  7. Add a second DVD drive
  8. Attach the Hyper-V (v1) Linux IC ISO to the second DVD drive of the VM
  9. Boot the VM into the Live CD and log in to its console
    • Debian will auto logon as ‘user’.
  10. switch to root:  sudo –i
    • This is specific to Debian Live
  11. Discover the IDE disks:
    • cd /dev
    • ls hd*
  12. Mount the virtual disk (the vhd)
    • make a mount point folder:  mkdir /mnt/mine
    • mount the disk to the folder:  mount /dev/hda1 /mnt/mine
  13. explore the volume
    • cd /mnt/mine
    • ls
    • Mine looks like a /boot volume:
    • clip_image001
  14. Mount the Linux IC DVD drive (mine is the second dvd on controller 2):
    • mkdir /mnt/cdrom
    • mount /dev/hdd /mnt/cdrom
  15. Copy the kernel shims from the ISO to the virtual disk
    • cd /mnt/cdrom/shim
    • cp *.* /mnt/mine
  16. Edit the device.map
    • cd /mnt/mine/grub
    • nano device.map
    • Before: clip_image002
    • After: clip_image003
  17. Edit the GRUB bootloader to load the shim and the kernel.
    1. nano menu.lst
    2. comment the ‘hiddenmenu’ option and increase the timeout so I can test.
      • clip_image004
    3. Create a new entry specific to the shim and the distribution kernel-xen
      • Notice that the kernel is the shim copied from the previous step and the existing kernel and initrd load as modules of the shim.
      • clip_image005
    4. Modify the default selection to point to my new entry.
      • The default entry begins counting at “0”
      • clip_image006
  18. Unmount the virtual disk and the cdrom
    • cd /
    • umount /dev/hda1
    • umount /dev/hdd
  19. Shutdown the virtual machine and remove the ISOs from the DVD drive (also remove the second virtual dvd drive).
  20. Boot the virtual machine, note the new menu selection that was created – this is the kernel that should boot.
    • clip_image007
Note: If you run VMware – This will not run on VMware, the shim is specific to Hyper-V.

Wednesday, August 19, 2009

Thinking in Workloads with OVF

Many of you realize that I am pretty close to the Citrix Project Kensho OVF Tool.

Frankly, I find it as a very useful tool with some very useful features.

First of all – let me mention a bit about OVF again.  OVF is NOT a method of converting virtual machines.  OVF is a way to describe a virtual appliance using standardized XML tags, so that various platforms can consume and use that virtual appliance as it was defined.

A virtual appliance has traditionally been though of as a single virtual machine (thank you VMware).  However, a virtual appliance is actually a “workload.”

Many of you might realize that an enterprise application is rarely a simple single .exe file that simply runs on a desktop.  A very simple reporting application might be an executable, a SQL database, and even a document archiving system.  All of these entities grouped together is the workload.

It takes all of these pieces working together for the application to be fully functional and feature rich.  The Application Workload would be a better way to describe this.

In the same light there is a component that might participate in multiple workloads – the SQL server can serve databases to multiple front-end and back-end applications.  It would have the most complex relationship in this example.

This brings me pack to the virtual appliance – the OVF is a description of the workload.  This example has that defined as two servers and one client.

If you are the person creating the package, you might leave the client out of the package, or only deliver the client executable as a component of the package, but it is not imported to a virtualization host as a virtual machine.

Some might call this creative thinking, but really it is just taking what the OVF allows and applying that to real situations.

The OVF standard (VMAN at the DMTF) is still evolving and changing.  And vendors are still working on compatibility and pushing those standards to ever complex designs.

It is because of this that not all vendors support each other.  They have to choose to allow for consumption of other implementations of the OVF standard.  Yes, this gets very complex and interwoven and creates a bummer for some folks that see OVF as the answer to virtual machine portability – when that portability has far more to do with the applications and operating systems within the virtual appliance themselves than it does in the depths of an XML file.

Wednesday, July 1, 2009

Linux vm from VMware to XenServer the videos

If you have been following, you will note quite a few posts related to importing / migrating Linux virtual machines from VMware to XenServer.

I realize that many folks don’t use XenServer – but the basic steps of repairing after migration apply to Hyper-V just as well as to XenServer – the steps are the same if you want your VM to boot. However, PV enablement on Hyper-V does not exist, you just need to install the vm tools.

Here are the links in case you missed them:

I have turned three of these into short (less than 10 minutes) video presentations, just to add a bit more information than in the articles.

Wednesday, June 17, 2009

XenConvert 2.0.1 is released with VMware OVF compatibility

We have been working on adding Citrix Project Kensho OVF capabilities to XenConvert.

XenConvert is the free Citrix machine conversion utility.  It is primarily focused on converting workloads to either Provisioning Server or to XenServer, however there are some more generic functions that are of interest to most any virtualization folk.

The download can be found here:

http://www.citrix.com/English/ss/downloads/details.asp?downloadId=1855017&productId=683148

If this moves in the future go here: http://www.citrix.com/English/ss/downloads/results.asp?productID=683148 and look for XenConvert in the XenServer download section.

OVF packages from any existing VMware product (known to this date) can be consumed (imported) direct to XenServer.

The physical to OVF path can be run within a Windows machine and convert it to an OVF (meta file + .vhd) or just a vhd.

The OVF can then be consumed to XenServer with XenConvert 2 or to XenServer and / or Hyper-V in the upcoming Kensho refresh.

The VHD can, of course, be copied to any engine that uses vhd.

It also does a binary conversion of vmdk to vhd and injects a critical boot device driver that is compatible with XenServer (and works with Hyper-V).

Also, the XenServer .xva (vm backup files) can be converted to OVF.

Download and enjoy!

Friday, May 22, 2009

The OVF and OVA dilemma

Here is a really good definition that I can thank my manager for of “OVF”:

OVF is a vendor- and platform-independent format to describe virtual machine metadata and create portable virtual-machine packages. These packages do not rely on a specific host platform, virtualization platform, or guest operating system. However, the virtual disk referenced by the OVF package must be compatible with the target hypervisor.

The Open Virtualization Format is a new and developing standard from the DMTF (Distributed Management Task Force). Its purpose is to form a common framework that any virtualization vendor can use to define a virtual workload. This workload can be simply a single virtual machine; this workload could also contain many virtual machines and also include complex descriptions for networking, applications, EULAs, and other entities. In theory, a single OVF could define all of the workloads within an entire enterprise and those workloads could be transported to a remote site for DR, or moved to a new datacenter. The possibilities are many.

The standard is here: http://www.dmtf.org/initiatives/vman_initiative/OVF_Tech_Note_Digital.pdf

The concept seems simple enough.

When talking to IT Professionals there is always time spent describing OVF and OVA, and where each fits.

“OVF” is the acronym for the standard.

What is referred to as an “OVF Package” is the collection of the workload states, plus a description meta file, plus any other entities that are defined in the description file.

BTW – the description file is also called the OVF file, or OVF meta file or OVF descriptor. This is why this gets really confusing really fast.

Here are the two important terms that get thrown around:

OVF / OVA – these are not the same thing.

An OVF is a collection of items in a single folder. Most commonly this is a description file (*.ovf) a manifest file (*.mf), and virtual machine state files (*.vhd or *.vmdk)

An OVA is a single file. The OVA is the OVF folder contents all zipped into a single file. The purpose of the OVA is when you want to take an OVF and share it, or give it as a download. The OVA needs to be opened into the OVF before it can be consumed.

Currently there are a host of folks working on OVF, you can learn more about the companies involved at the web site: http://www.dmtf.org/initiatives/vman_initiative/

Most commonly folks are running into OVF with Citrix (Project Kensho and XenConvert 2); VMware (VMware Workstation 6.5, Virtual Center 3 update 3, OvfTool, etc.); Sun (Virtual Box 2.2); and IBM. The big problem to date has been that the standards have been evolving during the time that these tools have been available. For example: a VMware v3 produced OVF is distinctly different than a VMware v4 OVF package. Mainly due to the state of the standards at the time the software was written. So, currently there is little cross platform between vendors.

The other problem that is happening right now is that defining a workload in an OVF is great, however, since these are virtual machine based – the operating system within the virtual machine has to deal with the fact that the virtualization hardware underneath it has been modified. Having a virtual machine as an OVF is not the same as having a virtual machine and running it through a conversion engine that repairs the operating system within the vm.

This is all evolving and will come in time.