Azure startup tasks run EVERY time your Role instance boots

I have recently had a total “DUH!” moment.  I almost called this post “The Revenge of the Batch Script.”

Azure has this feature called startup tasks.  These are task that run when your Roles instances boot.  And all the examples define these as command files (“.cmd” that is).  Okay, easy enough.

For some strange reason I have been stuck in the thinking that These run at instance provisioning; just after publishing my service.  As in these are RunOnce tasks.  But they actually run at the booting of an instance.  In fact they run anytime that any particular role instance boots.

It took me messing around with writing every single little thing to log files to really understand this.  And I also had to reboot my Azure Role instances when I was RDP’d into them.

What this really boils down to is that all those examples that you read about startup tasks where they mention to use them to install software; they are totally over simplified.  Your script needs to have some smarts, it needs to know if it ran before.  Because installing over a configured install doesn’t always work.  Reregistering a COM objects, doesn’t always work.

Oh, and if you use Azure Connect and define a an AD domain join – your VM will reboot as part of that domain join.  It is this precise thing that brought me here in the first place.

Here is my example of a startup task script with some very simple intelligence:

IF NOT EXIST %Public%\Documents\completedSetup.txt goto 20
ECHO The file completedSetup.txt exists.  Exiting..
GOTO END

:20
ECHO The file completedSetup.txt does not exist.  Setting up MyApp..

Perform all of my complicated application installation sequencing here..

Here I setup a task and execute it.  It is a PowerShell script that does my heavy lifting.  When it finishes it writes out the file that I am looking for.

:END
REM send a clean exit code
exit /b 0

Now, the key part lies in my PowerShell script.  At the end of my PowerShell script is:

Out-File $env:Public\Documents\completedSetup.txt

Something really simple, a little flag file, right where any user security context can find it.

PowerShell to add SSL binding to Default Web Site

Azure can be a bit perplexing at times – especially for those of us that are not developers.

This is where we all look at the VM Role as a way to get services running in Azure that have really complex set-ups.

At the same time, we can use Azure features to do part of the work for us.

Let me take the following scenario:  You have an application, this application requires specific IIS Role features, then some long application install, and last of all it has an involved set-up.  The scripting is all worked out, but it takes a really long time and there are a couple settings that have to be manually applied.

Your goal; automate as much as absolutely possible.

In Azure, you can add an SSL certificate to your Service – the Azure Service Certificate.

You can then configure Azure to automatically inject this certificate (public + private key) into your VM after it completes mini-setup in the Azure cloud.

Great!  I can use a certificate in my VM Role and the private key is not broken by the fact that I had to generalize the image with sysprep.  Now what?  I need to actually use this certificate.

I have two methods that I have worked out to add this certificate to the IIS default web site.

First of all – I created a self-signed certificate but I named it using my Azure Service DNS.  So the URL for my site is HTTP://MyService.CloudApp.Net 

(Don’t get all funny, you cannot purchase a cloudapp.net SSL certificate as this is Microsoft’s domain – you have to get your own domain, get your certificate and forward the DNS to your Azure Service URL).

Both of these methods I run within a PowerShell script – the different is how I achieve the results.

The common part before either method runs:

I had told Azure to put my certificate in the Local Machine\Personal store in the definition file:

After the VM exits mini-setup and my script runs I need to go find my certificate:

# Get all of the installed Local Computer certificates that can be used
# With Azure this is a Service Certificate that is defined in the configuration for this Role.

$allCerts = Get-ChildItem cert:\localmachine\my

# Find the Certificate that is defined for the domain name
foreach ($cert in $allCerts) {
    if ($cert.SubjectName.Name -match "CloudApp.Net") {
        $cloudCert = $cert
    }
}

Now, I have the certificate, let’s create the binding in IIS

Method 1 – the netsh way using apphost (Server 2008):

## Older Method to achieve the same thing.
$certThumb = $cloudCert.Thumbprint
## Generate a GUID for the SSL binding
$sslGuid = "{" + ([System.Guid]::NewGuid().toString()) + "}"
## Import the SSL certificate
netsh http add sslcert ipport=0.0.0.0:443 certhash=$certThumb appid=$sslGuid
## Create the SSL binding on port 443 for the Default Web Site
& "$env:windir\system32\inetsrv\appcmd.exe" set config -section:system.applicationHost/sites /+"[name='Default Web Site'].bindings.[protocol='https',bindingInformation='*:443:']" /commit:apphost

Method 2 – the PowerShell IIS module way (Server 2008 R2):

Import-Module WebAdministration

New-WebBinding -Name "Default Web Site" -IP "*" -Port 443 -Protocol https

$certObj = Get-Item $cloudCert.PSPath
New-Item IIS:SslBindings\0.0.0.0!443 -value $certObj

# Remove the HTTP binding
Remove-WebBinding -Name “Default Web Site” -BindingInformation *:80:

The new PowerShell 2.0 WebAdministration module way is much easier to deal with – but, it is PowerShell v2 – that is Server 2008 R2 and newer only.

Unable to ping Server 2008

I have blogged about this before and I will again as folks fall trap to this all of the time.

By default, ping response is disabled beginning with Server 2008 and continuing into newer revisions of Windows Server. 

FYI - With Server Core, ping is silent regardless of the state of the firewall.

Here is a great tip from Ben of the Hyper-V team:

The first thing I would do is to check the firewall configuration in your guest operating system, and make sure that it allows ICMP.

Here is a great article that steps you through how to do this:

“Nobody can ping my computer”

http://technet.microsoft.com/en-us/library/cc749323(WS.10).aspx

Sneaking around the lack of name resolution in Azure

This is one of those things that you will most likely run into if you try to run a traditional enterprise application in Azure – there is no name resolution.

Over many years enterprise applications have gone through the evolution of moving from IP addresses to relying on WINS to now DNS.  This move to using machine names provides great flexibility to server administrators to replace boxes at will, simply by editing a DNS entry or renaming the new server to the old server.  No need to update the configuration of the application.

Well, now we have Azure.  And although it is not Infrastructure as a Service, if you use a VM Role for some component of a historic enterprise application you do expect familiar features.

One way to handle this is to use Azure Connect.  Connect provides name resolution for the machine that participate in the virtual network.  However you only get IPv6 endpoint addresses back.  And this is useful for machine to machine communication but not for much else.

Another hitch is that in Azure machine names change.  Since all of the images are prepared with sysprep they come out of sysprep with new machine names (each provisioned instance needs to be unique, right?).  All that I know is that my worker tier machine will get new names if something happens and an instance gets replaced.  Or is someone stops my service and then starts it again, or reimages one of my instances.  With this going on, name resolution actually doesn’t help me much.

Well, I had a situation where I absolutely needed name resolution.  And that is one piece of information that I cannot query through the Azure Service Runtime.  So I have a homegrown solution – I edit the HOSTS file on each server.

In my example I only need to know the machine name of all of the instances of a particular role – so I query that to get the IP addresses.  I then turn around and use WinRM to query those instances and get the DNSHostName back from them.

I am using WinRM and not WMI because it is a single, incoming, well known port.  So I only have to define one endpoint at the Role level.  Through a bit of searching I discovered that WMI is not so fixed and I didn’t want to make it so.

Here is how it goes.  Warning:  I am not securing any of this.  I use HTTP for WinRM, and I reduce the PowerShell script execution security.  If you need to be secure and tight, then you need to tighten that up.

In my sysprep unattend.xml I first change the PowerShell execution policy.  I then run a script to set the WinRM service and client.  I then have a wait to give each instance a change to set the WinRM settings.  Then I query the Azure Service Runtime for the role instances and their IP addresses, then through WinRM I touch each server and get its DNSHostName.  Last, I append the local HOSTS file adding this information.  This executes on each server – they are all clones of each other after all.

Don’t forget to add the Internal Endpoint of 5985 to the Role definition in your service.  If you don’t, none of this matters.  If you secure this, use the endpoint of 5986.

Setting WinRM on each client (unsecurely, be aware of that):

<#
.SYNOPSIS
    A script to set the WinRM firewall rules and to configure the service and the
    client.  This enables unsecured communication.
.DESCRIPTION
    This script is designed to simply set up WinRM for remote unsecured communication.
    QuickConfig opens the firewall, the service settings allow remote connections,
    the client setting enables unsecured calling from the client to the remote service.
.LEGAL
    SCRIPT PROVIDED "AS IS" WITH NO WARRANTIES OR GUARANTEES OF ANY KIND, INCLUDING BUT NOT LIMITED TO
    MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE.  ALL RISKS OF DAMAGE REMAINS WITH THE USER, EVEN IF THE AUTHOR,
    SUPPLIER OR DISTRIBUTOR HAS BEEN ADVISED OF THE POSSIBILITY OF ANY SUCH DAMAGE.  IF YOUR STATE DOES NOT PERMIT THE COMPLETE
    LIMITATION OF LIABILITY, THEN DELETE THIS FILE SINCE YOU ARE NOW PROHIBITED TO HAVE IT.  TEST ON NON-PRODUCTION SERVERS.
.AUTHOR
    Brian Ehlert, Citrix Labs, Redmond, WA, USA
.REFERENCES
    Thank you TechNet. For examples. And a random forum post for the single quote fix.
#>

winrm quickconfig -quiet
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'
winrm set winrm/config/client '@{AllowUnencrypted="true"}'
winrm set winrm/config/client '@{TrustedHosts="*"}'

Now, to enumerate the information from the service runtime and append the HOSTS file:

<#
.SYNOPSIS
    A script to set the HOSTS file for Azure VMs to allow proper name resolution.
.DESCRIPTION
    In an Azure environment name resolution might not be available or might resolve IPv6 addresses.
    The Azure RuntimeService is queried to discover the IP addresses of other role members.  And then
    WinRM is used to query the DNSHostName of the other servers.
    The results are then appended to the HOSTS file.
.LEGAL
    SCRIPT PROVIDED "AS IS" WITH NO WARRANTIES OR GUARANTEES OF ANY KIND, INCLUDING BUT NOT LIMITED TO
    MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE.  ALL RISKS OF DAMAGE REMAINS WITH THE USER, EVEN IF THE AUTHOR,
    SUPPLIER OR DISTRIBUTOR HAS BEEN ADVISED OF THE POSSIBILITY OF ANY SUCH DAMAGE.  IF YOUR STATE DOES NOT PERMIT THE COMPLETE
    LIMITATION OF LIABILITY, THEN DELETE THIS FILE SINCE YOU ARE NOW PROHIBITED TO HAVE IT.  TEST ON NON-PRODUCTION SERVERS.
.AUTHOR
    Brian Ehlert, Citrix Labs, Redmond, WA, USA
.REFERENCES
    Thank you to Jason Fossen (http://blogs.sans.org/windows-security/). And to TechNet. For examples.
#>

# this is a local administrator and user name that is established in the VM
# Azure RDP access will automatically create / inject a user account that is defined
# Otherwise you need to establish a user account using your unattend.xml
$userName = "administrator"
$password = "Citrix`$2"
# Note:  I am using a plain text password.

# Add the Service Runtime snap-in to the standard Windows PowerShell command shell.
add-pssnapin microsoft.windowsazure.serviceruntime

# Take the VM Instance offline with Azure
Set-RoleInstanceStatus -Busy

$HostsFilePath = "$env:systemroot\system32\drivers\etc\hosts"

# Test the Hosts file by adding LocalHost entries
"127.0.0.1 localhost"  | add-content $HostsFilePath -force
"::1 localhost"  | add-content $HostsFilePath -force
if (-not $?) { "Error writing to hosts file!" ; return }

# Discover the other members of the Role
# It is not possible to have a server discover the Role that it is; as is the question "what am I" - it must be hardcoded.
# Enumerate all of the instances of the role named MyRole
$roleMem = Get-RoleInstance -Role MyRole

# Discover the endpoint IP the service is not necessary as it is the same for all
foreach ($roleIn in $roleMem) {

    # Find the WinRM port number
    foreach ($roleInEnd in $roleIn.InstanceEndpoints.Values){
        if ($roleInEnd.IPEndpoint.Port.Equals(5985)){
            #Get the IP of the endpoint
            $endIp = $roleInEnd.IPEndpoint.Address.ToString()
        }
        else{}
    }
   
    # winrm get wmi/root/cimv2/Win32_ComputerSystem (I simply find is easier to treat this as XML)
    $remoteServer = [xml](winrm enum wmi/root/cimv2/Win32_ComputerSystem -r:$endIp -encoding:utf-8 -a:basic -u:$userName -p:$password -format:pretty)
   
    # Add the entry to the HOSTS file
    $endIp + "`t" + $remoteServer.Results.Win32_ComputerSystem.DNSHostName
    $endIp + "`t " + $remoteServer.Results.Win32_ComputerSystem.DNSHostName |  add-content $HostsFilePath -force
   
}

That is it.  I use two scripts with a wait sequence in between (I use my random sleep script a couple articles back) since each machine is independent and must be able to call out to the others – the timing for WinRM happening early is important.

Getting Role Instance endpoint information from within an Azure VM

In a previous post I discussed the concepts of VM to VM communication in Azure.  The basics of how it works and where the knobs are to enable it.

Here is a little different focus.  It is the scenario that the endpoints have been defined, but your setup script that is installing or configuring your application needs to discover other roles and information about them because it needs to talk to them.

This is IPv4 traffic.  Most any application should know how to use this. 

I mentioned previously that name resolution is not available.  The other caveat is that I am not using Azure Connect (the Azure Virtual Network feature).  Connect gives name resolution, but it only gives me the IPv6 endpoint of the Connect VPN tunnel it does not give me the IP actual network IP addresses that my application wants.

So, PowerShell to the rescue.  To get you started I have this little PowerShell script that walks through my service, all Roles, and documents the endpoints that are enabled.

I run this on Instance A and I know where the open ports are configured on instance B.  I also use this to document my environment from within my environment by directing the output to a text file.

By the way, this is made possible by the Azure Service Runtime.  This is installed in VM Role VMs when the Windows Azure Integration Components are installed.  The hitch is that the VM must be in Azure for it to work, and you must be executing as Administrator.

<#
.SYNOPSIS
    A script to query the Azure Service Runtime and dump the endpoint configuration
    of all roles to a TXT file.
.DESCRIPTION
    This script is designed as a very simple troubleshooting tool for your VMs in Azure.
    It performs the very simple task of working through all Roles and Instances and
    writing the EndPoint configurations to a TXT file.
    This way when you are troubleshooting issues you have a quick document in the VM
    that you can reference to see what Internal EndPoints have been opened to
    facilitate VM to VM communication.
.LEGAL
    SCRIPT PROVIDED "AS IS" WITH NO WARRANTIES OR GUARANTEES OF ANY KIND, INCLUDING BUT NOT LIMITED TO
    MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE.  ALL RISKS OF DAMAGE REMAINS WITH THE USER, EVEN IF THE AUTHOR,
    SUPPLIER OR DISTRIBUTOR HAS BEEN ADVISED OF THE POSSIBILITY OF ANY SUCH DAMAGE.  IF YOUR STATE DOES NOT PERMIT THE COMPLETE
    LIMITATION OF LIABILITY, THEN DELETE THIS FILE SINCE YOU ARE NOW PROHIBITED TO HAVE IT.  TEST ON NON-PRODUCTION SERVERS.
.AUTHOR
    Brian Ehlert, Citrix Labs, Redmond, WA, USA
.REFERENCES
    Thank you TechNet. For examples.
#>

# declare the DumpInstance Function
function DumpInstance {
    param ($roleMem)
   
    foreach ($roleIn in $roleMem) {
    $i = ($roleIn.InstanceEndpoints.Count -  1)
    $roleIn.Role.Name + ", " + $roleIn.Id

    do {
    $keyName = $roleIn.InstanceEndpoints.Keys[$i]
    $endProtocol = $roleIn.InstanceEndpoints.Values[$i].Protocol
    $endPort = $roleIn.InstanceEndpoints.Values[$i].IPEndPoint.Port
    $endIp = $roleIn.InstanceEndpoints.Values[$i].IPEndPoint.Address.ToString()
    $endFamily = $roleIn.InstanceEndpoints.Values[$i].IPEndPoint.AddressFamily
   
    $endIp + ", " + $endPort + ", " + $keyName + ", " + $endProtocol + ", " + $endFamily
    --$i
    }
    until ($i -lt 0)
    ""
    }
}

# Add the Service Runtime snap-in to the standard Windows PowerShell command shell.
add-pssnapin microsoft.windowsazure.serviceruntime

# Get all Roles
$allRoles = Get-RoleInstance

foreach ($roleMem in $allRoles) {
    DumpInstance $roleMem
}