Friday, August 15, 2014

Getting errors from the Azure VM customer script extension without RDP

Since Azure has begun adding the VM agent and other extension to IaaS virtual machines (persistent VM Roles) a number of scenarios and possibilities have opened up.

The extension is a simple binary that is dropped into your VM and built to be triggered and to perform very specific actions in the security context of Local System.

The Desired State Configuration Azure extension is the very latest. 

Prior to this I have been spending some time with the Custom Script Extension.  And it is rather nifty.  But the biggest pain that I have had is in working through the process of troubleshooting the script as I develop it.

I have found no way to capture the standard output - other than directing to a text file.  But then I have to RDP into the VM to fetch it.

I can also look at the runtime status of the extension while connected over RDP - that that is one file with bad line endings, making it difficult to read in Notepad.

Trough a bit of searching I came across a few tips and started poking around a bit with the Azure PowerShell cmdlets. 

What I discovered is that you cannot get the standard output, but you can get the standard error through the API.  So, if the script tosses some terminating error, there is output to be fetched. If there was no error, there is no output to be returned.

What I ended up doing is the following:

New-AzureVM -Location $location -VM $vmConfig -ServiceName $Service

(Get-Date -Format s) + " .. Watch the script extionsion to monitor the deployment and configuration"
Do {
    $Vm = Get-AzureVM -Name $vmName -ServiceName $Service

    Get-Date -Format s
    "  Machine status: " + $Vm.Status
    "  Guest agent status: " + $Vm.GuestAgentStatus.FormattedMessage.Message
    foreach ( $extension in $Vm.ResourceExtensionStatusList ) {
        If ( $extension.HandlerName -match 'CustomScriptExtension' ) {
            "  ExtensionStatus: " + $extension.ExtensionSettingStatus.FormattedMessage.Message
            $scriptStatus = $extension.ExtensionSettingStatus.FormattedMessage.Message
            $scriptError = foreach ($substatus in $extension.ExtensionSettingStatus.SubStatusList) { ($substatus.FormattedMessage.Message).Replace("\n","`n") }
    Start-Sleep 10
} until ( ($scriptStatus -eq "Finished executing command") )

I fetch the VM, then drill into the object for the Custom Script Extension, then I dig into the Extension status, and it even has sub status.  It is in this sub status where the Standard Error ends up being bubbled up for the extension.

I realize that this leaves me waiting around and calling back and forth.  But a green light on "Finished executing command" only means the script extension is completed running whatever I told it to run, not that it worked.

I just wish I could get the standard output.


Friday, August 1, 2014

WAP Gallery Image, Dynamic IP address, and the SCVMM DHCP switch extension

Recently I had to put together a hands on lab for a number of sales engineers.
The lab involved SCVMM Service Templates, a custom Windows Azure Pack Gallery Image, and a Desired State Configuration module.

I had my environment of Hyper-V 2012 R2, SCVMM 2012 R2, and WAP about 95% configured.  As much as I could and still support the students re-using my VMs with their own Hyper-V Server.

Since the lab was not about WAP, but instead about my gallery image, I wanted to keep it as simple as possible.  I had a cloud, the cloud had a VM Network assigned, the students created a static IP pool.

(I already had an Internal Virtual Switch being created by SCVMM as a Logical Switch so that all lines of dependency were properly drawn)

In the WAP Admin portal - I had the students add the cloud and the VM Network to their plan.

I deploy my Gallery Image, and the domain join failed. 
I look closer, and I see that my VM ended up with an APIPA address and not an address from the IP Pool. 

Come to find out, the default behavior of a WAP Gallery Image is for dynamic IP address assignment. 
Which, if you only ever deploy a gallery image to a Windows Network Virtualization VM Network, you will never notice.  You will instead see that you get an IP from the IP Pool.

Something that I discovered long ago was that there is a custom Hyper-V Virtual Switch extension that ships with SCVMM.  It is actually a DHCP responder.  It catches the IP request, notifies SCVMM, and SCVMM responds with an IP from the SCVMM IP Pool assigned to the VM.  Nifty.

But, this path only happens if the VM is attached to a Windows Network Virtualization (NVGRE) network managed by SCVMM.

Back to the default Gallery Image behavior of a dynamic IP address.  No WNV network, no IP from an IP Pool.  How to fix this?

The only way to fix this is to open the Resource Definition of the Gallery Image, and then open the Network Profile, then the NIC.
And change the AllocationMethod to Static.

While you are in there, you will most likely notice a number of other interesting settings.

But the thing to be aware of is this, these are hard coded values, unless you work through making them settings that are actually exposed to your end customer (at this time you can't expose these settings).
If you change a setting here, that makes a dependency on an SCVMM placement rule, SCVMM will have to find a place that this VM can go to support all of the settings.  If it cannot, your VM will not be deployed.  And your tenant will call.

Tuesday, July 22, 2014

Copying files into Hyper-V VMs with Copy-VMFile

Over the life of Hyper-V there have been lots of convoluted ways that folks have used to get files in and out of Hyper-V VMs.

The most common method has been to mount the VHD and copy files in and out.  But you can't do this while the VM is running.

Then there the issue of using differencing disks or snapshots - and you want to replicate one file to many VMs.  Folks try and mount the parent virtual disk and copy files in - but due to the way that differencing disks work, this gives mixed results if it works at all.

Well, Hyper-V has a nifty feature of the Integration Components / Integration Services that allows you to inject files into a running VM.
The PowerShell cmdlet is Copy-VMFile.

I recently stumbled on this while getting some labs set up and I suddenly realized that I have 25 lab machines with 4 VMs each that my students will be using, and I have a broken lab if I don't correct one file.  Did I mention that I can't physically visit these servers?  I only have remote access.  What a pain.

Prior to being able to use the cmdlet you must have Guest Services enabled on your VM - and this is not on by default.

Enable-VMIntegrationService -Name 'Guest Service Interface' -VMName DSC01
Then, you can push a file into a VM from the Hyper-V Server by using -FileSource Host  And the Host is the only option.  You can only push in, not pullout.

Copy-VMFile -Name DSC01 -SourcePath .\ -DestinationPath 'C:\Users\Public\' -FileSource Host -CreateFullPath
You use the -Force parameter if you are overwriting an existing file.  And you don't need -Force otherwise.
-CreateFullPath does just what you would think it does, it creates the folder path you defined if it is not already present.

Simple as that.

There is some safety built into this I will mention.  Such as you cannot copy into the system path and other permissions blocks you will encounter.

Hyper-V has always approached the VM from the angle that it is evil, the VM is malicious.  This is the protection assumption.  Always keep that machine contained.

Wednesday, June 18, 2014

Cannot load Windows PowerShell snap-in microsoft.windowsazure.serviceruntime

There are some issues that simply lose interest due to not being pertinent enough, I believe I have found one.
And I can see from searching that I am not alone here.
The thing that is going on is that I am using startup tasks in a Worker Role to install an MSI and configure it.
When the script being executed by my startup task attempts to: add-pssnapin microsoft.windowsazure.serviceruntime

I receive the error:
add-pssnapin : Cannot load Windows PowerShell snap-in microsoft.windowsazure.serviceruntime because of the following error: The Windows PowerShell snap-in module
F:\plugins\RemoteAccess\Microsoft.WindowsAzure.ServiceRuntime.Commands.dll does not have the required Windows PowerShell snap-in strong name Microsoft.WindowsAzure.ServiceRuntime.Commands, Version=, Culture=neutral,
The work around that has been posted is to edit the registry key for the DLL, because the version is wrong.
My impression of this workaround is this:  "So, now I have to script a work around for an MSFT provided DLL because no one is updating the DLL registration for the snap-in in MSFT provided Gallery Images in Azure."
Mind you, these are developers, and if a workaround can be done in code - my experience has been that the issue is ignored after the workaround exists.

A workaround is great, but there are caveats to all of the workarounds posted thus far;
  • if I update the registry key with the incorrect version number, it still won’t load.  
  • I have to get the right version number and fix the registry entry.
  • And edit both the Assembly Name string and the version string.
Each posted workaround that I can find through search is a point version, and a point version edit.  A lasting workaround (since it seems like this bug is not going to get fixed anytime soon) is to dynamically deal with any version of the DLL and strong name enforcement in the OS.
I always preface that my solution might not be the most elegant thing that a developer comes up with, but I hope that folks can follow it and understand what I did.
# Get the current execution path
$exPath = Split-Path -parent $MyInvocation.MyCommand.Definition
$exPath # Echo for logging

### Before we can load the service runtime, we must fix the broken registry information for the DLL
### and the version must be right, or it will continue to be broken.

$runtimeDllReg = Get-Item HKLM:\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.WindowsAzure.ServiceRuntime
$runtimeDll = Get-Item $runtimeDllReg.GetValue('ModuleName')

$exportPath = $expath + "\AzureServiceRuntime.reg"

& $env:SystemRoot\System32\regedt32.exe /E $exportPath 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.WindowsAzure.ServiceRuntime'

# This is because the wait does not wait until regedit actually completes.

Do {

Start-Sleep 10

$regFile = Get-Item -Path $exportPath -ErrorAction SilentlyContinue

} Until ($regFile.Exists)

$runtimeRegFile Get-Content -Path $exportPath
$newRuntimeRegFile = @()

foreach ($e in $runtimeRegFile) {
     switch -wildcard ($e) {
          "**" { $e = $e -replace "", ($runtimeDll.VersionInfo.FileMajorPart.ToString() + '.' + $runtimeDll.VersionInfo.FileMinorPart.ToString() + '.0.0') }
     $newRuntimeRegFile += $e

Set-Content -Value $newRuntimeRegFile -Path $exportPath -Force

Start-Sleep 2

& $env:SystemRoot\System32\regedt32.exe /S $exportPath

Start-Sleep 2

# Load the necessary PowerShell modules for configuration and management Web and Work Roles
add-pssnapin microsoft.windowsazure.serviceruntime

# Take the VM Instance offline with Azure, or else Azure will keep attempting to start the script
Set-RoleInstanceStatus -Busy

You might wonder why I am using regedit.exe instead of the PowerShell cmdlets to set the value. What I found is that I kept running into permissions issues. And using regedit to export and import is a way around that.

Get more VM detail from Hyper-V

There was recently a question in the Hyper-V TechNet forum asking if there was a way to run the SCVMM Get-SCVirtualMachine cmdlet on a Hyper-V Server.

Well, technically yes;  you install the SCVMM Console PowerShell module on the Hyper-V Server - I don't think that was the intent of the question.

What the individual was looking for is more information about the VM - since SCVMM (by default) returns a ton of information about a virtual machine.
This is status, linked object detail, owner, etc.

Now, all of this information comes from the SCVMM Server, and much of this detail is unique to SCVMM.  At the same time, there is a lot more in Hyper-V than I think most folks realize, because it is filter output by default.

I am going to ignore SCVMM at the moment and focus on getting more from the in-box Hyper-V cmdlet.

Lets run through some examples:

PS C:\> Get-VM *
Name       State   CPUUsage(%) MemoryAssigned(M) Uptime     Status
----       -----   ----------- ----------------- ------     ------
WAPDev     Running 0           1580              4.20:34:51 Operating normally
myAdDns    Running 0           879               4.20:34:55 Operating normally
Downloader Off     0           0                 00:00:00   Operating normally
bjeSql     Running 0           2749              4.20:34:52 Operating normally

PS C:\> Get-VM WAPDev | Format-List

Name             : WAPDev
State            : Running
CpuUsage         : 0
MemoryAssigned   : 1656750080
MemoryDemand     : 1275068416
MemoryStatus     : OK
Uptime           : 4.20:37:01
Status           : Operating normally
ReplicationState : Replicating
Generation       : 2

Now, the individual in the forum post requested the following: "The SCVMM cmdlet Get-SCVirtualMachine returns a plethora of information regarding a VM. I would like to run this cmdlet on a Hyper-V host and get the same level of detail for each VM running on it. It appears that this is not possible."

Let me take my example one step further and show you one simple command that can give you everything about a VM.  It is a command the simply tells the PowerShell cmdlet to return all properties of the VM, not the default filtered set.  That is what the pipe to select * does.

And from here you can dig into the object and go off down some rabbit hole of dependencies such as NetworkAdapters, or the DVDDrives, or the HardDrives.

PS C:\> Get-VM WAPDev | Select *

VMName                      : WAPDev
VMId                        : 7ff0d548-1032-4cce-91ba-9a100a27b113
Id                          : 7ff0d548-1032-4cce-91ba-9a100a27b113
Name                        : WAPDev
State                       : Running
IntegrationServicesState    : Up to date
OperationalStatus           : {Ok}
PrimaryOperationalStatus    : Ok
SecondaryOperationalStatus  :
StatusDescriptions          : {Operating normally}
PrimaryStatusDescription    : Operating normally
SecondaryStatusDescription  :
Status                      : Operating normally
Heartbeat                   : OkApplicationsHealthy
ReplicationState            : Replicating
ReplicationHealth           : Normal
ReplicationMode             : Primary
CPUUsage                    : 0
MemoryAssigned              : 1656750080
MemoryDemand                : 1275068416
MemoryStatus                : OK
SmartPagingFileInUse        : False
Uptime                      : 4.20:39:47
IntegrationServicesVersion  : 6.3.9600.16384
ResourceMeteringEnabled     : False
ConfigurationLocation       : E:\WAPDev
SnapshotFileLocation        : E:\WAPDev
AutomaticStartAction        : StartIfRunning
AutomaticStopAction         : Save
AutomaticStartDelay         : 0
SmartPagingFilePath         : E:\WAPDev
NumaAligned                 : False
NumaNodesCount              : 1
NumaSocketCount             : 1
Key                         : Microsoft.HyperV.PowerShell.VirtualMachineObjectKey
IsDeleted                   : False
ComputerName                : SWEETUMS
Version                     : 5.0
Notes                       :
Generation                  : 2
Path                        : E:\WAPDev
CreationTime                : 11/22/2013 11:42:23 AM
IsClustered                 : False
SizeOfSystemFiles           : 69268
ParentSnapshotId            :
ParentSnapshotName          :
MemoryStartup               : 2147483648
DynamicMemoryEnabled        : True
MemoryMinimum               : 1073741824
MemoryMaximum               : 1099511627776
ProcessorCount              : 2
RemoteFxAdapter             :
NetworkAdapters             : {Network Adapter}
FibreChannelHostBusAdapters : {}
ComPort1                    : Microsoft.HyperV.PowerShell.VMComPort
ComPort2                    : Microsoft.HyperV.PowerShell.VMComPort
FloppyDrive                 :
DVDDrives                   : {DVD Drive on SCSI controller number 0 at location 1}
HardDrives                  : {Hard Drive on SCSI controller number 0 at location 0}
VMIntegrationService        : {Time Synchronization, Heartbeat, Key-Value Pair Exchange, Shutdown...}

So, the next time you want to know more about your VM, or you want to do a relationship walk from the VM object to the dependent object - just remember to "select *" and return all properties of the VM to you $vm object.  It takes longer, it works WMI harder, but it might get you where you are headed.

Once you know what you are looking for - using Get-VM, Get-VMMemory, Get-VMDVDDrive, Get-VMHardDisk, Get-VMSnapshots, etc. are actually faster - since they focus on the class of object and filter by that class.