SCVMM is deploying using the wrong VHD or VHDX

I like this one.  It is interesting.
This is a classic case of 'as designed' being exposed through changes in behavior as an implementation evolves.  Combined with what you see in the UI not really being what happens in the system.

I like it because I consider it a bug due to the fact that the behavior has changed between release and applying Update Rollups.

The symptom is this: you deploy a VM (or a bare metal deployment) and you observe that SCVMM is using the incorrect virtual disk.
And you noticed this happening consistently after applying UR2 (IMHO - that is the key differentiator in the behavior change).

In a nutshell - SCVMM is selecting the virtual disk from the Library.  And it is actually getting more than one from the Library and it is using the incorrect one.

Lets back up a bit.  How is SCVMM ending up with an incorrect virtual disk from the Library in the first place.

Objects in the SCVMM Library have this concept of equivalency.  Multiple objects being equivalent to each other, though they are different objects.
And virtual disks are one of these objects.

Lets say that you create one VHD and this is Server 2012 R2, sysprep'd, all ready to go.  You assign that to a Host Profile or a Service Template, or a WAP Gallery Item.  You set the version to 1.0.0.0 and the family to 'Server 2012 R2' and the OS to 'Server 2012 R2 Datacenter'.
You perform some test deployments and all is good, you move ahead and use it.

After a bit, you get new hardware and you create a new VHD.  Identical to the first except you add some additional drivers for the new storage cards and NICs.
You set the version and family and OS the same.

You then open the Host Profile or Service template and select this new disk.
You deploy, and ... your hardware does not work.  Your new device drivers are missing.  You jump up and down and bang your head on the wall.
You run around in circles.

What happened was this.  SCVMM didn't select the single object that you think you selected.  It selected the OS, family, and version.  And it got two back from the Library.  It then used the "first" one, not the second one.  (this is the behavior change - prior to UR2 it was as if the one with the later date was used, but no longer).

Make the version unique (or the family name) and you will fix the problem.

According to the VMM team - this is 'as designed'.  That is developer speak for:  We built it to work that way.
I hope the documentation will soon describe this equivalency concept.  Since it currently mentions it in passing and never really describes anything about it.

Until then, I hope search brings you here.

Uninstalling software from Server 2012 R2 Core

Back a while ago I had a short post about uninstalling the SCVMM agent from Server Core.

That was an older version of Server and an older version of PowerShell.

Today, I had a reason to revisit that process.  But now I have a bit more finesse with PowerShell and PowerShell is now in its v4 revision with Server 2012 R2.

And, needless to say, it is considerably simpler.

I am simply going to capture and paste my PowerShell screen and allow you to sort out the rest by reading through it.  I think you will understand it simply enough.

PS C:\> Get-WmiObject -Class win32_product | ft

IdentifyingNumber Name Vendor Version Caption
----------------- ---- ------ ------- -------
{1D8E6291-B0D5-35EC-... Microsoft Visual C++... Microsoft Corporation 10.0.40219 Microsoft Visual C++...
{3834A905-5CC1-454D-... Microsoft System Cen... Microsoft Corporation 3.2.7510.0 Microsoft System Cen...
{A89BBF08-D933-4634-... Microsoft System Cen... Microsoft Corporation 3.2.7620.0 Microsoft System Cen...


PS C:\> Get-WmiObject -Class win32_product


IdentifyingNumber : {1D8E6291-B0D5-35EC-8441-6616F567A0F7}
Name : Microsoft Visual C++ 2010 x64 Redistributable - 10.0.40219
Vendor : Microsoft Corporation
Version : 10.0.40219
Caption : Microsoft Visual C++ 2010 x64 Redistributable - 10.0.40219

IdentifyingNumber : {3834A905-5CC1-454D-8CA4-AC449F12775D}
Name : Microsoft System Center Virtual Machine Manager DHCP Server (x64)
Vendor : Microsoft Corporation
Version : 3.2.7510.0
Caption : Microsoft System Center Virtual Machine Manager DHCP Server (x64)

IdentifyingNumber : {A89BBF08-D933-4634-8FBB-EB88F870981B}
Name : Microsoft System Center Virtual Machine Manager Agent (x64)
Vendor : Microsoft Corporation
Version : 3.2.7620.0
Caption : Microsoft System Center Virtual Machine Manager Agent (x64)



PS C:\> $installed = Get-WmiObject -Class win32_product
PS C:\> $installed[1].Uninstall()


__GENUS : 2
__CLASS : __PARAMETERS
__SUPERCLASS :
__DYNASTY : __PARAMETERS
__RELPATH :
__PROPERTY_COUNT : 1
__DERIVATION : {}
__SERVER :
__NAMESPACE :
__PATH :
ReturnValue : 0
PSComputerName :



PS C:\> $installed[2].Uninstall()


__GENUS : 2
__CLASS : __PARAMETERS
__SUPERCLASS :
__DYNASTY : __PARAMETERS
__RELPATH :
__PROPERTY_COUNT : 1
__DERIVATION : {}
__SERVER :
__NAMESPACE :
__PATH :
ReturnValue : 0
PSComputerName :



PS C:\> Get-WmiObject -Class win32_product


IdentifyingNumber : {1D8E6291-B0D5-35EC-8441-6616F567A0F7}
Name : Microsoft Visual C++ 2010 x64 Redistributable - 10.0.40219
Vendor : Microsoft Corporation
Version : 10.0.40219
Caption : Microsoft Visual C++ 2010 x64 Redistributable - 10.0.40219



PS C:\>

Most useful Desired State Configuraiton debugging commands

Okay, here they are:

Get-ScheduledTask "Consistency" | Start-ScheduledTask

This will immediately re-process a configuration.

My scenario is this:
I place my configuration on a pull server.
I then configure the DSC Local Configuration Manager with the GUID of that configuration.

Then I can tweak the configuration and play with it, and not have to push it each time.
Running the consistency command allows me to 'do it now' and force a configuration to be applied, instead of waiting for my configured application window of time to pass.

Trace-xDscOperation | where { $_.EventType -eq "error" } | fl

This requires that you have the xDscDiagnostics module on your target machine. (Which if you are sorting out using DSC and applying configurations - this is necessary).
Lots of folks have blogged about it.
And it is handy with only two commands Get-xDscOperation and Trace-xDscOperation and the value add that it will enable tracing for you the first time you run in. Usability is high in this one.

Get-xDscOperation shows the job list.

Get-xDscOperation

This error you will always see the first time you apply a configuration.

If you want to know if DSC is chugging along and doing its thing you have to check the status of the consistency job:

Get-ScheduledTask "Consistency"

TaskPath                                       TaskName    State
--------                                       --------    -----
\Microsoft\Windows\Desired State Configurat... Consistency Running

Now, lets look at another failure to see the details.

Trace-xDscOperation | where { $_.EventType -eq "error" } | fl

There is good detail in here, you just have to find it.

[Script]MediaDownload is one of my configuration items. It is the Script Resource Provider (that is necessary context for the next)

The detail of the error is actually: MSFT_ScriptResource failed to execute Set-TargetResource functionality with error message: Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (404) Not Found." .

This tells me that the file I am attempting to download is missing (or my URI is wrong).

Another useful case for Trace-xDscConfiguration is right after you apply a configuration using Set-DscLocalConfigurationManager.

When I do this I frequently find a comment that the task will be created pending a reboot.  This indicates to me that DSC is going to do absolutely nothing until I reboot that machine.

I thought I was going insane the first time nothing happened and I stumbled on this message - you don't see it if you use the -Verbose switch, only in the trace.

Domain join credential formatting for Windows Azure Pack Gallery Items

I normally don’t blog about something that I consider to be a bug, but in this case the failure behavior is difficult to piece together so I thought I would.

Windows Azure Pack Gallery items have the ability to have multiple credentials defined within them.  And these are used for various actions within the application scripts that are defined within the Gallery Item.

There are two credentials are are essentially ‘built in’ – the local administrator and the domain join user.

If you use the VM Role Author – these credentials are defined automatically.  If you spend time playing around with the WAP Tenant API – you will see that these credentials are labeled as ‘intrinsic’ settings on the object.

The local administrator account you cannot avoid – this sets the local administrator password on the VM and it is required by WAP.  The user ‘administrator’ is grayed out, so you have to set the password for the local administrator.

The domain join user is something you cannot avoid if you define that your VM will join a domain.

Your Gallery Item can have additional user credentials as well.  Say a special one that is used to configure an application, or you have a script that adds a user to the local administrators group, or you need to perform some action against a remote SQL Server and need the proper user credentials.

Now – defining user accounts – there are two format options:  domain\username and username@domain.

Well guess what – for the domain join username, you can only use the domain\username format.  If you attempt to use username@domain you will see that the The provisioning of the VM fails. 

The failure message in the SCVMM job log is:

Warning (22044)

One or more virtual machines have failed during customization during the deployment of the service.

Nothing is clear until I look at the unattended.xml that is generated by the SCVMM deployment process to apply to the VM.

The one that works is:

            <Identification>

                <JoinDomain>global.local</JoinDomain>

                <Credentials>

                    <Domain>global</Domain>

                    <Username>administrator</Username>

                    <Password>********</Password>

                </Credentials>

            </Identification>

The one that fails is:

               <JoinDomain>global.local</JoinDomain>

                <Credentials>

                    <Domain xsi:nil="true" />

                    <Username>administrator@global.local</Username>

                    <Password>********</Password>

                </Credentials>

Does this error also apply if a user types the username@domain format in the GUI for the domain join credential when deploying my Gallery Item?  I just tried it – the username@domain format also fails in the GUI. 

Now, I mentioned other credentials – not the ‘special’ credential that is the domain join user.  Can these credentials be defined using username@domain?  Yes, yes they can.  Those credentials can be defined using either format.

Setting Hyper-V default paths and migration settings with Desired State Configuration

Okay, now we are getting useful.  Base configuration with Desired State Configuration.

But I will tell you, when I first looked at the modules I immediately wanted to configure default settings of the Hyper-V Server itself and that was totally lacking.

So I decided to spend a few days and create cVMHost – not all of the settings that I want, but enough to cover my needs on my test lab.  You will find it on the PowerShell.org Git repository under DSC.

With it I open the following:

• Configure default VHD path
• Configure default Virtual Machine path
• Configure VM Migration
• Configure enhanced session mode

There are a few fail-safe features I added.  Such as testing the paths, and validating that Hyper-V is there in case the dependencies are left out of the configuration.  At the same time, I didn’t enable defining the migration network or the security model.  I had to start with something useful, and I do have a day job so I could not make it too large.

So, just the xHyper-V – you download the cHyper-V and unzip it to C:\Program Files\WindowsPowerShell\Modules so that PowerShell will automatically load it.

To use it, here is a sample configuration that builds on my other two posts:

configuration Sample_cVMHost
{ 

    WindowsFeature hypervRole
    {
        Ensure = 'Present'
        Name = 'Hyper-V'
    }
    WindowsFeature hypervManagement
    {
        Ensure = 'Present'
        Name = 'Hyper-V-PowerShell'
        DependsOn = '[WindowsFeature]hypervRole'

    } 

    Import-DscResource -module cHyper-V
    PSHOrg_cVMHost hostSettings
    {
        VMHost = 'localhost'       
        Ensure = 'Present'
        VirtualDiskPath = 'C:\users\public\VHDs'
        VirtualMachinePath = 'C:\users\public\VMConfig'
        VirtualMachineMigration = $true
        EnhancedSessionMode = $false
        DependsOn = '[WindowsFeature]hypervManagement'
    } 

    Import-DscResource -module xHyper-V

        xVMSwitch ExternalSwitch
        {
            Ensure         = 'Present'
            Name           = 'VMs'
            Type           = 'External'
            NetAdapterName = (Get-NetAdapter)[0].Name
            AllowManagementOS = $true 
        }
} 

Sample_cVMHost

Start-DscConfiguration -Wait -Verbose -Path .\Sample_cVMHost

And there you have it, you have migration enabled, and you could enable enhanced session mode, and you have new default storage paths since you could have an SMB share, or iSCSI storge or some big LUN sitting there that you want to default to.

And if you really want to get sneaky.  Modify these settings prior to creating your VMs using the other xHyper-V modules.