XenDesktop Windows Azure Pack Gallery Image Tech Preview

Today we launched our XenDesktop 7.5 Windows Azure Pack Gallery Image Tech Preview as a download from Citrix.com.

(Yea!)

This is open to customers and prospective customers and is intended to simplify and automate XenDesktop deployments for large enterprises and service providers who leverage Windows Azure Pack, System Center, and the Microsoft Cloud OS stack.

If you are not already familiar; a Windows Azure Pack Gallery Image is a standard, reusable, and sharable artifact that allows customers of a large enterprise or service provider to self-serve provision virtual machine roles as a repeatable configuration.

The current XenDesktop specific Gallery Image can installs all XenDesktop roles.

https://www.citrix.com/downloads/xendesktop/betas-and-tech-previews/xendesktop-75-windows-azure-pack-gallery-image-tech-preview

We are also looking for feedback regarding where to take this next. So as always, you can add comments here, or find me on Twitter, or email me directly. 

This is a continuation of similar efforts to streamline XenDesktop deployments that we rolled out with the XenDesktop System Center Templates launched for XenDesktop 7.1 late last year and for XenDesktop 7.5 in the last couple of months.

 

WAP Gallery Image, Dynamic IP address, and the SCVMM DHCP switch extension

Recently I had to put together a hands on lab for a number of sales engineers.
The lab involved SCVMM Service Templates, a custom Windows Azure Pack Gallery Image, and a Desired State Configuration module.

I had my environment of Hyper-V 2012 R2, SCVMM 2012 R2, and WAP about 95% configured.  As much as I could and still support the students re-using my VMs with their own Hyper-V Server.

Since the lab was not about WAP, but instead about my gallery image, I wanted to keep it as simple as possible.  I had a cloud, the cloud had a VM Network assigned, the students created a static IP pool.

(I already had an Internal Virtual Switch being created by SCVMM as a Logical Switch so that all lines of dependency were properly drawn)

In the WAP Admin portal - I had the students add the cloud and the VM Network to their plan.

I deploy my Gallery Image, and the domain join failed. 
I look closer, and I see that my VM ended up with an APIPA address and not an address from the IP Pool. 

Come to find out, the default behavior of a WAP Gallery Image is for dynamic IP address assignment. 
Which, if you only ever deploy a gallery image to a Windows Network Virtualization VM Network, you will never notice.  You will instead see that you get an IP from the IP Pool.

Something that I discovered long ago was that there is a custom Hyper-V Virtual Switch extension that ships with SCVMM.  It is actually a DHCP responder.  It catches the IP request, notifies SCVMM, and SCVMM responds with an IP from the SCVMM IP Pool assigned to the VM.  Nifty.

But, this path only happens if the VM is attached to a Windows Network Virtualization (NVGRE) network managed by SCVMM.

Back to the default Gallery Image behavior of a dynamic IP address.  No WNV network, no IP from an IP Pool.  How to fix this?

The only way to fix this is to open the Resource Definition of the Gallery Image, and then open the Network Profile, then the NIC.
And change the AllocationMethod to Static.

While you are in there, you will most likely notice a number of other interesting settings.

But the thing to be aware of is this, these are hard coded values, unless you work through making them settings that are actually exposed to your end customer (at this time you can't expose these settings).
If you change a setting here, that makes a dependency on an SCVMM placement rule, SCVMM will have to find a place that this VM can go to support all of the settings.  If it cannot, your VM will not be deployed.  And your tenant will call.

SCVMM is deploying using the wrong VHD or VHDX

I like this one.  It is interesting.
This is a classic case of 'as designed' being exposed through changes in behavior as an implementation evolves.  Combined with what you see in the UI not really being what happens in the system.

I like it because I consider it a bug due to the fact that the behavior has changed between release and applying Update Rollups.

The symptom is this: you deploy a VM (or a bare metal deployment) and you observe that SCVMM is using the incorrect virtual disk.
And you noticed this happening consistently after applying UR2 (IMHO - that is the key differentiator in the behavior change).

In a nutshell - SCVMM is selecting the virtual disk from the Library.  And it is actually getting more than one from the Library and it is using the incorrect one.

Lets back up a bit.  How is SCVMM ending up with an incorrect virtual disk from the Library in the first place.

Objects in the SCVMM Library have this concept of equivalency.  Multiple objects being equivalent to each other, though they are different objects.
And virtual disks are one of these objects.

Lets say that you create one VHD and this is Server 2012 R2, sysprep'd, all ready to go.  You assign that to a Host Profile or a Service Template, or a WAP Gallery Item.  You set the version to 1.0.0.0 and the family to 'Server 2012 R2' and the OS to 'Server 2012 R2 Datacenter'.
You perform some test deployments and all is good, you move ahead and use it.

After a bit, you get new hardware and you create a new VHD.  Identical to the first except you add some additional drivers for the new storage cards and NICs.
You set the version and family and OS the same.

You then open the Host Profile or Service template and select this new disk.
You deploy, and ... your hardware does not work.  Your new device drivers are missing.  You jump up and down and bang your head on the wall.
You run around in circles.

What happened was this.  SCVMM didn't select the single object that you think you selected.  It selected the OS, family, and version.  And it got two back from the Library.  It then used the "first" one, not the second one.  (this is the behavior change - prior to UR2 it was as if the one with the later date was used, but no longer).

Make the version unique (or the family name) and you will fix the problem.

According to the VMM team - this is 'as designed'.  That is developer speak for:  We built it to work that way.
I hope the documentation will soon describe this equivalency concept.  Since it currently mentions it in passing and never really describes anything about it.

Until then, I hope search brings you here.

Domain join credential formatting for Windows Azure Pack Gallery Items

I normally don’t blog about something that I consider to be a bug, but in this case the failure behavior is difficult to piece together so I thought I would.

Windows Azure Pack Gallery items have the ability to have multiple credentials defined within them.  And these are used for various actions within the application scripts that are defined within the Gallery Item.

There are two credentials are are essentially ‘built in’ – the local administrator and the domain join user.

If you use the VM Role Author – these credentials are defined automatically.  If you spend time playing around with the WAP Tenant API – you will see that these credentials are labeled as ‘intrinsic’ settings on the object.

The local administrator account you cannot avoid – this sets the local administrator password on the VM and it is required by WAP.  The user ‘administrator’ is grayed out, so you have to set the password for the local administrator.

The domain join user is something you cannot avoid if you define that your VM will join a domain.

Your Gallery Item can have additional user credentials as well.  Say a special one that is used to configure an application, or you have a script that adds a user to the local administrators group, or you need to perform some action against a remote SQL Server and need the proper user credentials.

Now – defining user accounts – there are two format options:  domain\username and username@domain.

Well guess what – for the domain join username, you can only use the domain\username format.  If you attempt to use username@domain you will see that the The provisioning of the VM fails. 

The failure message in the SCVMM job log is:

Warning (22044)

One or more virtual machines have failed during customization during the deployment of the service.

Nothing is clear until I look at the unattended.xml that is generated by the SCVMM deployment process to apply to the VM.

The one that works is:

            <Identification>

                <JoinDomain>global.local</JoinDomain>

                <Credentials>

                    <Domain>global</Domain>

                    <Username>administrator</Username>

                    <Password>********</Password>

                </Credentials>

            </Identification>

The one that fails is:

               <JoinDomain>global.local</JoinDomain>

                <Credentials>

                    <Domain xsi:nil="true" />

                    <Username>administrator@global.local</Username>

                    <Password>********</Password>

                </Credentials>

Does this error also apply if a user types the username@domain format in the GUI for the domain join credential when deploying my Gallery Item?  I just tried it – the username@domain format also fails in the GUI. 

Now, I mentioned other credentials – not the ‘special’ credential that is the domain join user.  Can these credentials be defined using username@domain?  Yes, yes they can.  Those credentials can be defined using either format.

SCVMM Service deployment Error (22758) The system cannot find the file specified – but it is right there

I spend days on this issue and did not want to lose the ‘why’ this cost me days.

First, the details of the error:

Error (22753)

The script command with properties: Type (PreInstall), Deployment Order (20) and Parent Type (ApplicationProfile), failed to complete successfully. Refer to the errors list for more information.

Error (22758)

The guest agent encountered an exception while running a script command. Win32ErrorCode: 2147942402 DetailedErrorMessage: The system cannot find the file specified

I can definitively tell you that this error has nothing and something to do with the script being run.  And, this is the SCVMM Agent itself tossing this error, not my script.

First; the most useful log in SCVMM Service Templates is the SCVMM Guest Agent diagnostic log.  You find it buried in the OS of the VM under %ProgramFiles%\<System Center Version>\<VMM Guest Agent Version>\bin\diagnostics\VmmGuestLog.svclog

If you try to open this up with Notepad or WordPad or NotePad++ it just looks like a bunch of poorly formatted XML.  Very difficult to troll through.

However, if you use Microsoft Service Trace Viewer, you have a totally different experience.  Trace Viewer is part of Visual Studio, which you can get an Express version for free, or download it here.

Looking through this I go looking for the events that the SCVMM Job log tossed at me.

I find:

4: GetExpandedGCEErrorInfo reported The script command with properties: Type (PreInstall), Deployment Order (20) and Parent Type (ApplicationProfile), failed to complete successfully. Refer to the errors list for more information.

But not the Win32ErrorCode.

Right after the above is the following:

4: Encountered an exception for ActionItem Type: Gce, Exception: Microsoft.VirtualManager.GuestAgent.Common.GuestAgentException: Error in the application.
   at Microsoft.VirtualManager.GuestAgent.Gce.GceProcessor.GetStdErrorMatch()
   at Microsoft.VirtualManager.GuestAgent.Gce.GceProcessor.PerformPostProcessing(List`1& warnings)
   at Microsoft.VirtualManager.GuestAgent.Gce.GceProcessor.Run(IQueryCancel queryCancel, List`1& warnings)
   at Microsoft.VirtualManager.GuestAgent.AIFactory.GceActionProcessor.ExecuteActionItemSync(Boolean restarting, ActionItem actionItem, IQueryCancel queryCancel, List`1& warnings)
   at Microsoft.VirtualManager.GuestAgent.QueueManager.QueueManager.QueueProcessorThread(Object context)

This definitively tells me that it is not my script, but instead the SCVMM Agent is the one not finding the file. The best that I can determine is that that GetStdErrorMatch() being empty is the root.  But what does a positive return look like?

And, to make it worse, the script ran.  Everything worked!  And if I checked the VM all of the SCVMM artifacts from my Custom Resource were there ( %ProgramData%VirtualMachineManagerData\<one of the temp paths>).  Nothing was missing!

I spent the next few days changing the script, settings for the execution, accounts, paths, different ways to execute it.  It would work sometimes but mostly not.  This was the most frustrating part.  Every now and then I would not get the error; that small glimmer of hope and gets you frustrated and ends up driving you insane.

On the drive home last night the pieces slid together.  This was my log cleanup script.  It was deleting all of the log files that were empty (0kb).  I produce a lot of logs,  this is remote and headless execution, it could be happening anywhere but I have no console to observe anything.  You have to capture every little thing that happens and you don’t want to sort through 26 files when only the 5 with any details actually mean anything.

Not being the one that wrote the code what I am guessing is that; It appears that the output file is created, then fetched, then written back to. And my script deleted it in the middle somewhere.  Simply timing alone, milliseconds, would let it pass.  A moment of a slow disk drive on a hypervisor, possibly.

So, I removed the capturing of the Standard Out and Standard Error and all is good.  Seems like a silly little thing, right?  But again, this is remote installation, if you don’t log it you have no idea what happened.  So turning off logging was actually the very last thing I had any desire to do.