Getting errors from the Azure VM custom script extension without RDP

Since Azure has begun adding the VM agent and other extension to IaaS virtual machines (persistent VM Roles) a number of scenarios and possibilities have opened up.

The extension is a simple binary that is dropped into your VM and built to be triggered and to perform very specific actions in the security context of Local System.

The Desired State Configuration Azure extension is the very latest. 

Prior to this I have been spending some time with the Custom Script Extension.  And it is rather nifty.  But the biggest pain that I have had is in working through the process of troubleshooting the script as I develop it.

I have found no way to capture the standard output - other than directing to a text file.  But then I have to RDP into the VM to fetch it.

I can also look at the runtime status of the extension while connected over RDP - that that is one file with bad line endings, making it difficult to read in Notepad.

Trough a bit of searching I came across a few tips and started poking around a bit with the Azure PowerShell cmdlets. 

What I discovered is that you cannot get the standard output, but you can get the standard error through the API.  So, if the script tosses some terminating error, there is output to be fetched. If there was no error, there is no output to be returned.

What I ended up doing is the following:

New-AzureVM -Location $location -VM $vmConfig -ServiceName $Service

(Get-Date -Format s) + " .. Watch the script extionsion to monitor the deployment and configuration"
Do {
    $Vm = Get-AzureVM -Name $vmName -ServiceName $Service
    Get-Date -Format s
    "  Machine status: " + $Vm.Status
    "  Guest agent status: " + $Vm.GuestAgentStatus.FormattedMessage.Message
       
    foreach ( $extension in $Vm.ResourceExtensionStatusList ) {
        If ( $extension.HandlerName -match 'CustomScriptExtension' ) {
            "  ExtensionStatus: " + $extension.ExtensionSettingStatus.FormattedMessage.Message
            $scriptStatus = $extension.ExtensionSettingStatus.FormattedMessage.Message
            $scriptError = foreach ($substatus in $extension.ExtensionSettingStatus.SubStatusList) { ($substatus.FormattedMessage.Message).Replace("\n","`n") }
        }
    }
    Start-Sleep 10
} until ( ($scriptStatus -eq "Finished executing command") )

I fetch the VM, then drill into the object for the Custom Script Extension, then I dig into the Extension status, and it even has sub status.  It is in this sub status where the Standard Error ends up being bubbled up for the extension.

I realize that this leaves me waiting around and calling back and forth.  But a green light on "Finished executing command" only means the script extension is completed running whatever I told it to run, not that it worked.

I just wish I could get the standard output.




 

My Azure Public Endpoint switcher

I recently worked through an issue with the Persistent VM beta in Windows Azure around remotely managing my VMs (machines as I will generically call them).

Now, my scenario is that I have multiple machines in Azure behind a single Service (DNS name at cloudapp.net).  You can of course bring your own DNS name and simply mask this, you are not stuck with it.

In order to get a remote session to each one, only one can have the RDP or SSH port at any time for the Service. 

So the setup resembles this:

• brianEhApp.cloudapp.net:3389 -> VM1:3389
• brianEhApp.cloudapp.net:56789 -> VM2:3389
• brianEhApp.cloudapp.net:54321 -> VM3:3389

This is relatively straightforward.  However, I ran into a situation what I was not able to authenticate to the higher ‘ephemeral’ ports that Azure was automatically generating as the administrative endpoint for my +1 machines.

So, I could use the Portal to change one VM then back out to the other and so on.

However, this was taking quite a long time.  And I am impatient.

So, I turned over to the Azure PowerShell cmdlets and whipped out the following script.  It switches the public endpoints for me and launches my administrative RDP session straight away.  Considerably faster than messing with the Portal.

$myVm = Read-Host -Prompt "what is the VM name you want to access via RDP?"
$myService = Read-Host -Prompt "what is the Service name where the VM resides?"
"The assumption is that the endpoint name is RDP"

# Check all the endpoints of all the VMs and move off 3389 to set desired VM to 3389
$deploymentVms = @()
$deploymentVms = Get-AzureVM -ServiceName $myService 

# First all VMs in the Service need to be off 3389 or there will be an error setting the correct one
foreach ($vm in $deploymentVms){
    $vmEnds = @()
    $vmEnds = Get-AzureEndpoint -VM $vm
    Foreach ($end in $vmEnds){
        If ($end.Port -eq 3389 -and $vm.Name -notmatch $myVm){
            $pubPort = Get-Random -Minimum 49152 -Maximum 65500
            Set-AzureEndpoint -Protocol tcp -LocalPort 3389 -PublicPort $pubPort -Name $end.Name -VM $vm | Update-AzureVM
        }
    }

}

# Now, set the desired VM Public endpoint to 3389
Get-AzureVM -ServiceName $myService -Name $myVm | Set-AzureEndpoint -Protocol tcp -LocalPort 3389 -PublicPort 3389 -Name RDP | Update-AzureVM

Get-AzureRemoteDesktopFile -ServiceName $myService -Name $myVm -Launch

Azure Virtual Machine the PowerShell Basics

Really, I am not going to go into a lot of explanation here.  I am assuming that you have some PowerShell background and need a bit to get started.  That is what I am covering.

You can find the cmdlets here:

https://www.windowsazure.com/en-us/manage/downloads/

Be sure to have your Azure management certificate properly stored in your Personal certificate store prior to connecting to your subscription.

These first commands are pretty much mandatory when you begin a PowerShell session.

Import the module:

​import-module 'C:\Program Files (x86)\Microsoft SDKs\Windows Azure\PowerShell\Azure\Azure.psd1'

Import a settings file (this speeds up as it lists all subscriptions you have access to - to create this file perform Export-AzurePublishSettingsFile (Visual Studio also uses this))

Import-AzurePublishSettingsFile 'C:\Users\Public\Documents\BrianEhServices.publishsettings'

Choose the subscription that you will interact with for your session:

Select-AzureSubscription -SubscriptionName "Sample Subscription"

Set the default Storage account that will be used (it must be in the same subscription)

Set-AzureSubscription -SubscriptionName "Sample Subscription" –CurrentStorageAccount SampleStorageAccount

By the way, do the above is handy.  Like me I assume that you all have alt least two Azure subscriptions (yours and the one your company has given you access to).  Using that settings file allows easy switching.  Now that you have that, go exploring.

If you have a Service or have deployed a Virtual Machine from the Gallery using the Portal you can query it, change it, dispose of it.

If you have VHD images you can manipulate those.  Now, disclaimer here, my experience so far is that CSUPLOAD from the Azure SDK is still the best way to get VHDs into Azure storage and it now supports both stateless and persistent VHD images.  It differentiates between the two because it registers them with the VHD repository for you and sets its life in motion.

But, I assume that you are getting itchy, so lets just begin with making a new Virtual Machine in a new Service.  The very same thing that you would get if you used a Gallery VM image (this is not the Quick VM as that would be New-AzureQuickVM).

First we need to find an image:

List all available images: Get-AzureVMImage

List all available in a table: Get-AzureVMImage | Format-Table

Find images that have been uploaded to your Storage account ('user' images): Get-AzureVMImage | where { ($_.Category -eq "user") }

Now I just want to use the Server 2012 Gallery image and create a VM (we will build on this command).

$svr2012Image = Get-AzureVMImage | where { ($_.Category -eq "Microsoft") -and ($_.Label -match "Server 2012" ) -and ($_.ImageName -match "Datacenter") }

Apply a customization configuration to the image:

$myImage = New-AzureVMConfig -Name MyNewMachine -InstanceSize ExtraSmall -ImageName $svr2012Image.ImageName
Add-AzureProvisioningConfig -VM $myImage -Windows –Password <complex password>

Create the Service and Provision the VM:

New-AzureVM -ServiceName "MyNewService" –VMs $myImage

Notice after it is finished that Azure automatically created an RDP endpoint to allow remote OS access (if it was a Linux image an SSH endpoint would have been created).

Azure IaaS Virtual Machine concepts

The Azure IaaS Virtual Machine is in Beta and anyone can request access.

For those have worked with Azure VMs in the past (Web, Worker, and VM Role) this is simply an addition, a new option.

There are some interesting contrasts, parallels, and dependencies when we consider both the PaaS and IaaS options (MSFT likes to draw the contrast that way, but fundamentally they are still machines what you can do and need to know is different).

MSFT already has lots of documents out there on the IaaS stuff.  I don’t intend to regurgitate what they have already documented.  They stuff they have not documented is far more interesting.

And there is a gallery of OS images that you can build machine from using the GUI portal.  And you can create and upload your own VHD images (a bit like VM Role).

In my next post I will begin exploring the PowerShell that is available to drive this stuff from the ground, that is a bit more interesting (at least to me).

First of all, what is the Virtual Machine?  If it isn’t a Web, Worker, or VM Role; what is it.

It is a bit like a VM Role, but without any agents to interact with the Azure Fabric, and the storage supports persistence.  Beyond persistence, it is just a machine.  Mind you, the way I state this it does not sound like much, but it took a lot for MSFT to work through extrapolating the PaaS provisioning and storage into supporting this.  There is a lot there.  Just go look up the TechEd talks.

Now, we all know that MSFT al about platforms, so the Virtual Machine by itself is just as uninteresting as it sounds.  Add in Virtual Networks and it gets a bit more interesting.

Virtual Networks allow defining of address allocation spaces.  And those can be divided into subnets.  If you know the SCVMM IP Pool model, it is similar to that with some implementation differences.

A Virtual Network is also a container or a boundary.  All machines within a Virtual Network can freely communicate with each other over the network. No matter the IP that was allocated to the VM, no IP subnet isolation.  A Virtual Network can also span services – provided they are in the same Affinity group and Region.  It all links.

A Virtual Network also supports the Gateway.  The Gateway is a way to ling the Virtual Network defined in Azure with an SSL VPN endpoint on the ground, in your enterprise.  This is just like any point-to-point VPN that you have used to connect branch offices, or to another business.  Before this we only had Azure Connect and its support for IPv6 and that it only acted as a machine to machine forwarder, fundamentally different.

One other concept is the VHD.  There are two types.  Image and OS. 

An Image can be provisioned into multiple VMs and is prepared so that Azure can customize it.  The Windows OS is prepared with sysprep, and the Linux OS is prepared with the Azure Linux Agent deprovision command.  The concept is the same.

When a new VM is created from an image it can and will be customized.  And, the Fabric also automatically creates a management endpoint (SSH or RDP).

An OS can be used by one Virtual Machine.  It is not provisioned, and therefore not customized.  It is just created an powered on.  You have to define everything, but it allows you to bring your own, pre-configured machine into Azure (without building it in space).

In the next few posts we will get a bit more into this and specifically driving it all with PowerShell.

The last concept is the Services.  IaaS Virtual Machines still run within a Service just like PaaS VMs.  But the two shall not mix.

You can create a Service without a deployment, this is androgynous, lacking either PaaS or IaaS gender.  As soon as the deployment is defined the service becomes either IaaS (meaning that it can run Virtual Machines) or PaaS (meaning that it can run VMs (or Roles)).  Thus the entire Service becomes stateless or persistent.

The two can interact over the network, but not through Azure Fabric Services.

I think that in the next post I am going to dive straight into the PowerShell.