Hyper-V Resource Pool Introduction

There is a feature of Hyper-V 2012 that is rarely discussed but is highly useful.

First, the Resource Pool concept; 

For this I pulled a definition from the Office documentation; “A set of resources that is available for assignment”  that is the best way that I can describe it.

Second, this is not VMware style resource pools.  Their implementation is very unique.  It is closer to the XenServer implementation of resource pools.  However, it does not follow that either.

If you head over to the DMTF and search on Resource Pool you find something nice and vague about Resource Pool Hierarchies.  Okay, we will see that in the implementation.

So, before I move on, those of you with VMware backgrounds, just forget using resource pools to manage reservations and what not.  It might eventually get there, but not today.

As I mentioned in the Office quote, the Resource Pools represent assignment.  A connection.  A relationship (DMTF). 

If you want to see an easy example of the Resource Pools in action (and in the GUI), you need to create one.

The example

Lets look at Networking.  Connecting VMs, enabling VMs for all kinds of migrations, and inconsistent configurations.

I have two Hyper-V Servers.  They were set up by different folks, with different naming preferences.  Joe names his virtual network “VMs” and Gale names hers “LAN”.  Each time they move a VM back and forth they need to reconfigure the network settings of the VM. 

There has to be a way to do that without renaming their virtual switches.  There is.  Create an “Ethernet” Resource Pool.

Okay, now some PowerShell and some details

PS C:\Users\Administrator> get-command *resourcepool*

CommandType     Name                                               ModuleName
-----------     ----                                               ----------
Cmdlet          Get-VMResourcePool                                 Hyper-V
Cmdlet          Measure-VMResourcePool                             Hyper-V
Cmdlet          New-VMResourcePool                                 Hyper-V
Cmdlet          Remove-VMResourcePool                              Hyper-V
Cmdlet          Rename-VMResourcePool                              Hyper-V
Cmdlet          Set-VMResourcePool                                 Hyper-V

If you have done nothing with Resource Pools on a Hyper-V Server and you simply type Get-VMResourcePool you actually get a bunch back.

PS C:\Users\Administrator> Get-VMResourcePool

Name       ResourcePoolType       ParentName ResourceMeteringEnabled
----       ----------------       ---------- -----------------------
Primordial FibreChannelConnection            False
Primordial FibreChannelPort                  False
Primordial ISO                               False
Primordial VFD                               False
Primordial VHD                               False
Primordial Ethernet                          False
Primordial Memory                            False
Primordial Processor                         False

These are the Primordial Resource Pools.  Hyper-V gives these to you because you can only add children, the Server must provide the first Parent.  In this case they called it Primordial.  Sounds all medieval doesn’t it?  It simply means there is nothing before, it is definitely the Root, not some Pool that someone called “root”.

So, lets make a new Resource Pool and attach it to the Ethernet Primordial Pool.

PS C:\Users\Administrator> New-VMResourcePool -Name "VM LAN" -ResourcePoolType Ethernet

Name   ResourcePoolType ParentName   ResourceMeteringEnabled
----   ---------------- ----------   -----------------------
VM LAN Ethernet         {Primordial} False

Notice how the Primordial pool was assumed in this case?  Handy.

Now, I could have explicitly defined –ParentName and I probably would have if I had multiples.  Because I could branch Resource Pools if I wanted to. 

If I branch Resource Pools I can use them to create logical groupings for metering or connecting devices.  And each one would have different options because it is at a different level and combination of parents.

This familial stuff can get pretty messy so I will keep this example simple.

Now that I have my Resource Pool.  How do I use it?

Well, now that I created a Resource Pool. If I open the settings of the Network Adapter for any VM on that Hyper-V Server I see something totally new.

And if I drop the selection list I can connect to the Primordial ( “<Root>” ) or the Resource Pool I just created.

If I select the Pool I just created the Virtual Switch setting is changed.

Because I didn't associate a Virtual Switch with my new Resource Pool.  This command is not totally intuitive, I expect to use Set-VMSwitch to modify the setting, however, the clever PM behind the Hyper-V cmdlets decided to use a different verb.

PS C:\Users\Administrator> Add-VMSwitch -ResourcePoolName "VM LAN" -Name VMs

Now, If I open the settings of the VM again.  It makes sense to select “automatic connection” for the switch.  This way the Pool is connected to, not the switch.

I can actually name the switch in some way unique to the server or hardware, and have the consistent naming abstracted above that.  So, whatever switch is associated with this Pool, the VM will be connected to it.

What else can I do with that Resource Pool?

Well, I can Measure it of course.

First, enable Metering then measure it.

PS C:\Users\Administrator> Enable-VMResourceMetering -ResourcePoolType Ethernet -Name "VM LAN"
PS C:\Users\Administrator> Measure-VMResourcePool -ResourcePoolType Ethernet -Name "VM LAN"

Name   ResourcePoolType AvgCPU(MHz) AvgRAM(M) TotalDisk(M) NetworkInbound(M) NetworkOutbound(M)
----   ---------------- ----------- --------- ------------ ----------------- ------------------
VM LAN {Ethernet}                                          0                 0

Hmm, seems that no time passed, so there is no data.  Waiting a bit, lets Measure again.

PS C:\Users\Administrator> Measure-VMResourcePool -ResourcePoolType Ethernet -Name "VM LAN"

Name   ResourcePoolType AvgCPU(MHz) AvgRAM(M) TotalDisk(M) NetworkInbound(M) NetworkOutbound(M)
----   ---------------- ----------- --------- ------------ ----------------- ------------------
VM LAN {Ethernet}                                          2                 1

Hey, some network traffic.  Excellent.

This is just an introduction to Resource Pools.  I hope to bring some more in the future as they are highly useful, yet relatively invisible.

My Windows Network Virtualization demo setup script

In the past I posted my two Hyper-V Server demo script for Windows Network Virtualization.

And a whole series of posts about Windows Network Virtualization.  Unfortunately, some folks just could not get it working.

Well, a key script that I left out was the one I used to setup my servers.  And that is here.

I use the little trick of setting the IP of the VMs through WMI as well (remember, the VMs must be running).

I also use a little trick to build all of the routes and rules.  Remember, there is no automatic route maintenance, if you are setting this up and using it, you must also manage the change – the system won’t do it for you.

One requirement here, the machines are joined to the same domain.  And I have a Register-PSSessionConfiguration on the remote machine defined as Taylor mentions here.

Okay, enough of that.  Here is my setup and reset PowerShell script:

 

$what = Read-Host -Prompt "Do you want to RESET or SETUP?"

# Load the function to set the IP address.
function Get-VMIPAddress ($VMName)
{
    $Query = "SELECT * FROM Msvm_ComputerSystem WHERE ElementName = '" + $VMName + "'"
    $VM = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    if ($VM -eq $null)
    {
        write-host ("No VM found with the specified name '" + $VMName + "'")
        return
    }

    $Query = "ASSOCIATORS OF {" + $VM.__PATH + "} WHERE ResultClass = Msvm_VirtualSystemSettingData"
    $VMSettings = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    $Query = "ASSOCIATORS OF {" + $VMSettings.__PATH + "} WHERE ResultClass = Msvm_SyntheticEthernetPortSettingData"
    $VMNIC = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    $Query = "ASSOCIATORS OF {" + $VMNIC.__PATH + "} WHERE ResultClass = Msvm_GuestNetworkAdapterConfiguration"
    $VMIpSettings = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    $VMIpSettings
}

function Set-VMIPAddress ($VMName, $IPSettings)
{
    $Service = Get-WmiObject -Class "Msvm_VirtualSystemManagementService" -Namespace "root\virtualization\v2"

    $Query = "SELECT * FROM Msvm_ComputerSystem WHERE ElementName = '" + $VMName + "'"
    $VM = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    if ($VM -eq $null)
    {
        write-host ("No VM found with the specified name '" + $VMName + "'")
        return
    }

    $Value = $Service.SetGuestNetworkAdapterConfiguration($VM, $IPSettings.GetText(1))

    #Value success if the return value is "0"
    if ($Value.ReturnValue -eq 0)
    {
        write-host "Operation completed successfully"
    }

    #If the return value is not "0" or "4096" then the operation failed
    elseif ($Value.ReturnValue -ne 4096)
    {
        write-host ("Operation failed: " + $Value.ReturnValue)
    }

    else
    {
        #Get the job object
        $job = [WMI]$Value.job

        if ($job.JobState -eq 7)
        {
            # Job has already failed

            write-host "Operation failed"
            write-host "ErrorCode:" $job.ErrorCode
            write-host "ErrorDescription" $job.ErrorDescription
        }
        else
        {
            #Provide updates if the jobstate is "3" (starting) or "4" (running)
            while ($job.JobState -eq 3 -or $job.JobState -eq 4)
            {
                write-host $job.PercentComplete
                start-sleep 1

                #Refresh the job object
                $job = [WMI]$Value.job

                #A jobstate of "7" means success
                if ($job.JobState -eq 7)
                {
                    write-host "Operation completed successfully"
                }
                else
                {
                    write-host "Operation failed"
                    write-host "ErrorCode:" $job.ErrorCode
                    write-host "ErrorDescription" $job.ErrorDescription
                }
            }
        }
    }
}

switch ($what) {
    RESET {
        "RESET!!"
        # Get set up and make sure that it is all ready

        Invoke-Command -ComputerName Waldorf.brianeh.local -ConfigurationName HVRemoteAdmin { Move-VM -Name “Red22” -DestinationHost Statler.brianeh.local –IncludeStorage –DestinationStoragePath D:\BlankVM }

        sleep 30

        # Reset adapters
        Get-VMNetworkAdapter -VMName * | Connect-VMNetworkAdapter -SwitchName VMs
        Get-VMNetworkAdapter -VMName * | Set-VMNetworkAdapter -VirtualSubnetId 0

        Get-VMNetworkAdapter * | ft vmname, name, macaddress, ipaddresses, virtualsubnetid

        # Reset the VMs to DHCP
        # Red11
        $VMIP = Get-VMIPAddress "Red11"
        $VMIP.DHCPEnabled = $True
        Set-VMIPAddress "Red11" $VMIP

        # Red22
        $VMIP = Get-VMIPAddress "Red22"
        $VMIP.DHCPEnabled = $True
        Set-VMIPAddress "Red22" $VMIP

        # Blue11
        $VMIP = Get-VMIPAddress "Blue11"
        $VMIP.DHCPEnabled = $True
        Set-VMIPAddress "Blue11" $VMIP

        # Blue22
        $VMIP = Get-VMIPAddress "Blue22"
        $VMIP.DHCPEnabled = $True
        Set-VMIPAddress "Blue22" $VMIP

        # Reset the Windows NetVirt binding

        $vSwitch = Get-VMSwitch -SwitchType External
        Disable-NetAdapterBinding -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription -ComponentID "ms_netwnv"

        Invoke-Command -ComputerName waldorf.brianeh.local -ConfigurationName HVRemoteAdmin {
            $vSwitch = Get-VMSwitch -SwitchType External
            Disable-NetAdapterBinding -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription -ComponentID "ms_netwnv"
        }

        # power off VMs, Red first
        Stop-VM Red* -AsJob
        sleep 60
        Stop-VM Blue* -AsJob

    }

    SETUP {
        "SETUP!!"
        # power on VMs, Red first
        Start-VM Red* -AsJob
        sleep 60
        Start-VM Blue* -AsJob

        # wait until all VMs ICs are 'awake' and are reporting an IP address
        $vmNics = Get-VMNetworkAdapter *
        do {
            $upCount = ($vmNics | where {$_.IPAddresses.count -ne 0})
            $upCount.Count
            sleep 30
        }
        until ($vmNics.Count -eq $upCount.Count)

        # Red11
        # Set the IP of each VM
        $VMIP = Get-VMIPAddress "Red11"

        #Setting the VM Network settings to a static IPv4 address
        $VMIP.DNSServers = @("192.168.1.2")
        $VMIP.IPAddresses = @("192.168.1.11")
        $VMIP.DefaultGateways = @("192.168.1.1")
        $VMIP.Subnets = @("255.255.255.0")
        $VMIP.DHCPEnabled = $False

        Set-VMIPAddress "Red11" $VMIP

        # Red22
        $VMIP = Get-VMIPAddress "Red22"

        $VMIP.DNSServers = @("192.168.1.2")
        $VMIP.IPAddresses = @("192.168.1.22")
        $VMIP.DefaultGateways = @("192.168.1.1")
        $VMIP.Subnets = @("255.255.255.0")
        $VMIP.DHCPEnabled = $False

        Set-VMIPAddress "Red22" $VMIP

        # Blue11
        $VMIP = Get-VMIPAddress "Blue11"

        $VMIP.DNSServers = @("192.168.1.2")
        $VMIP.IPAddresses = @("192.168.1.11")
        $VMIP.DefaultGateways = @("192.168.1.1")
        $VMIP.Subnets = @("255.255.255.0")
        $VMIP.DHCPEnabled = $False

        Set-VMIPAddress "Blue11" $VMIP

        # Blue22
        $VMIP = Get-VMIPAddress "Blue22"

        $VMIP.DNSServers = @("192.168.1.2")
        $VMIP.IPAddresses = @("192.168.1.22")
        $VMIP.DefaultGateways = @("192.168.1.1")
        $VMIP.Subnets = @("255.255.255.0")
        $VMIP.DHCPEnabled = $False

        Set-VMIPAddress "Blue22" $VMIP

        Get-VMNetworkAdapter * | ft vmname, name, macaddress, ipaddresses, virtualsubnetid

    }
}

Securing Azure Virtual Machines or I got hacked and how you shouldn’t

I have been working with Azure for over a year now and this is the first time I have had a machine compromised.  I will say that this entire incident has been a good learning experience.  I think that anyone can learn from my mistakes here.

Thinking back this event is interesting from many angles.

• a default machine provisioning behavior is opening a door that you (as the customer) needs to remember to close or not open in the first place.
• the machine is really out there, in the wild, innocent, vulnerable.  Not protected in your datacenter behind all the other protections.
• your IT Security folks might not yet be engaged in how to deal with deploying into public clouds.
• you are doing test, exploration, or other work on systems that are not going to production.

 

Let’s go back in time to the week of November 26. 

My innocent Virtual Machine was compromised by an attack against the RDP protocol.  The well known RDP port 3389 was the vector.

The attacker succeeded in obtaining access, changing the local administrator password and pwning my box.  I can only image that it was then ‘sold’ on some botnet as later it ended up with a PayPal phishing site.

Then it began the process of being capacity to begin finding other vulnerable folks. This is where I noticed what was happening.

(BTW – the MSFT folks that handle this type of incident were extremely helpful in working out what happened, in what order, and how.)

 

First, lets look at the environment where I put my machine.

It was in Azure.  Placing any machine in Azure is equivalent to placing it in the corporate DMZ.  There is a load balancer between you and the outside world, and public endpoint definition(s) that define the public ingress port(s).

Beyond that, you can actually do bunches of different configurations of Virtual Networks, or Gateways.  You can link Services.  All kinds of stuff.  But remember, all IaaS machine to machine network traffic within a Service is wide open.  There are no private endpoints (as in a PaaS Service).

That said, any IT Pro would immediately say ‘duh’ or ‘you idiot’.  Yep.  Nothing I can comment there.

 

There are some default IaaS behaviors that are enabling this;

• automatically opening an RDP endpoint for management
• requiring complex passwords (false sense of security)
• local administrator is not disabled
• no real guidance around practices (not that anyone RTFM’s any longer anyway – they simply SearchTFM)

PaaS Roles don’t have these behaviors.  And the RDP connection is secured with a Certificate and a secondary user account.

Some of the Azure MVPs found my original MSDN forum thread and beat my blog post out.  My IT Pro background is leading me to some different ideas, and I felt I needed to tell a bigger story.

And, as I think about this; remember that you are administering these lockdowns at the same time that your machine could be getting probed, or brute force attacked.  So, choose a logical order, define a strong password at the point to machine creation (however you do it).  At least it gives you a fighting chance.

 

So, this is where i mention a bit of guidance.  Here are some options to pick and choose from.

Some initial mitigation suggestions to choose from:

• Use the PaaS roles of Web or Worker as the front end machines whenever possible.  They are already hardened.
• Rename the local administrator account.
• Disable the local administrator account and create some uncommonly named user account for administrative access.
• Choose strong plus complex passwords, or passphrases.  Not simply one or the other.  The OS can enforce complexity but not strength. 
• A dictionary attack is likely to hit “P@ssw0rd” but it is unlikely to hit “Just a city boy, born and raised in South Detroit”.
• My favorite example of password strength: http://xkcd.com/936/ 
• Denying user access after X failed logon attempts (lock the account).  This is a Local security policy if not domain joined, or a Domain policy if joined.  Consider an automatic (timed) unlock as well, or you could have no recourse but to destroy your machine.
• Do not allow the creation of the default RDP public endpoint.  This is only possible through the API / PowerShell.  Or delete the auto created endpoint after creating the machine in the Portal.
• Only create the RDP endpoint when remote administration is necessary, and removing it after.  But remember that we are human, and unless you have some interface doing this for you, you will probably forget at some point.
• Remove the RDP endpoint and use the Virtual Network Gateway feature of the Azure Virtual Network for secured remote administration without public endpoints.  This requires some ground based router, and the VPN is slow, but your ports are closed.
• Remove RDP endpoint & use Azure Connect.  This is limited to IPv6 TCP traffic only, but that should cover anything required to manage the OS.
• Avoid 3389 as the public port (I noticed my compromised machine specifically scanning for this port to spread itself) by using a port in the ephemeral range.
• Use the Windows Advanced Firewall rules and define them appropriately.
• Use Windows IP Security Policies and tightly define the sources from which RDP traffic can be accepted from.  This is highly effective, but a pain to set up.
• Monitor the machine.  Azure provides metrics through the portal and API.  Discover a baseline.  Use an agent within the machine.  This only detects the compromise after it happens and is not preventative.
• Take a snapshot of the clean state.  This is not a point and click thing in Azure today, but you can work this out using the Storage cmdlets through destroying your machine, making the diff disk, and reincarnating the machine.

 

Some of these practices are simply security through obscurity but, in this case, I think it’s OK.

• Changing the name of the administrator account dramatically increases the search space for a username/password pair.
• Changing the RDP endpoint is going to increase the search space for a TCP port/username/password trio.

I didn’t mention antivirus.  I won’t go into my theories there.  My personal opinion is that if all the tightening measures are done and you still get hacked, most likely your AV would have failed as well.

Always be ready for the possibility of rebuilding a machine.  Never assume you can simply move forward, or even install a backup.  The best you can do is revert to a known clean state.

 

Good luck!

Setting VM IP with Hyper-V WMI

There is a hidden feature in Server 2012.  Buried deep in the bowels of the WMI classes is this little tidbit that is discoverable, but not talked about.

You can set the IP of a VM through Hyper-V 2012. 

For a long time now, in the forums we have told folks that this cannot be done.  Well, it can be.  And at small scale it works.  I have used it to set up my Network Virtualization demonstration environment.

The thing that you need to be aware of is that this is one of those cases where you send your command to WMI / CIM and you need to double back and check what happened.  Did the IP actually set? 

Be sure to check that before going off and expecting that it did.  If it didn’t, you have event logs in the VM.  Check there.

This took a bit of working through and discovery.  I never did work through IPv6, I stopped when I got IPv4 working.

<speculation>

As you might imagine, there has to be a dependency on the aligning of the version of the Integration Services within the VM and the Hyper-V Server.  I could not imagine this working with Hyper-V 2012 and a Server 2008 VM when the Integration Components in the VM have not been updated to the current Hyper-V level.

</speculation>

One thing that I did not blog about with the Network Virtualization script was how I set up my environment, more on that next, I scripted it, and it is not small.

Back the the subject, setting the IP of a VM through WMI of the Hyper-V Server.  I am going to leave out all the error handling just to make this easier to read through.

Here is a PowerShell function where you can see the WMI in action:

function Set-VMIPAddress ($VMName, $IPSettings)

{

    $Service = Get-WmiObject -Class "Msvm_VirtualSystemManagementService" -Namespace "root\virtualization\v2"

    $Query = "SELECT * FROM Msvm_ComputerSystem WHERE ElementName = '" + $VMName + "'"

    $VM = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    $setIP = $Service.SetGuestNetworkAdapterConfiguration($VM, $IPSettings.GetText(1))

}

Of course, you want to consider $setIP.ReturnValue

Like all WMI / CIM commands if the return value is “0” then you have success, “4096” generally means it is running.  And anything else means something went wrong in most cases.

If you have “4096” then query for the status of the job itself.  $job = [WMI]$setIP.job

By the way, if you want to get the IP of the VM don’t think $Service.GetGuestNetworkAdapterConfiguration.  This is proper thinking for PowerShell but not for WMI / CIM.  CIM could never be that easy.  ;-)

Look for the class Msvm_GuestNetworkAdapterConfiguration to get the IP of a VM.  Find the VM NIC with Msvm_SyntheticEthernetPortSettingData first.

Paravirtulization under the hood and more

For those of you that are hard core virtualization folks, there is an excellent couple of articles over at xen.org by George Dunlap from Citrix.

In the ESX world and Hyper-V world the virtualization is closer to the HVM type or PVHVM when the OS is enlightened.  Xen has grown from a different root and started from the paravirutalization world (true PV, it is actually kind of interesting how the VMs themselves boot in this world).

This also gives a bit of background into the terminology and options that are available.

There is a part 1: http://blog.xen.org/index.php/2012/10/23/the-paravirtualization-spectrum-part-1-the-ends-of-the-spectrum/

and part 2: http://blog.xen.org/index.php/2012/10/31/the-paravirtualization-spectrum-part-2-from-poles-to-a-spectrum/

Personally, I think it good reading for anyone working with machines as it is a history of evolution in one aspect.

At this same time we have MSFT Research working on the Library OS.  This is an interesting abstraction of applications into VM type containers, application containers.  This is more similar to the traditional Xen PV model, where (technically) there isn’t a boot kernel in there, just the runtime components of the machine and the bootstrap comes from the xen hypervisor itself.  (at least that is my impression of it).

The MSFT research project known as Drawbridge: http://research.microsoft.com/en-us/projects/drawbridge/

And a bit more: http://research.microsoft.com/apps/pubs/default.aspx?id=141071

And a Channel9 presentation for the short attention spans among us:  http://channel9.msdn.com/Shows/Going+Deep/Drawbridge-An-Experimental-Library-Operating-System

Other MSFT Research OS projects: http://research.microsoft.com/en-us/groups/os/

Is this the future?  the Application level virtualization that was discussed many years ago.  Decoupling the application from the OS?  Not really the decoupling, but the forcing of an application into a container.  A container that it cannot get out of and affect other applications.

I look at this and think about traditional application compatibility issues going away, true application throttling, true isolation of a session (and its applications) within a Terminal Server.  That is what really makes me think about where this is all headed.  And we continue to be just at the beginning of it all.

My Azure Public Endpoint switcher

I recently worked through an issue with the Persistent VM beta in Windows Azure around remotely managing my VMs (machines as I will generically call them).

Now, my scenario is that I have multiple machines in Azure behind a single Service (DNS name at cloudapp.net).  You can of course bring your own DNS name and simply mask this, you are not stuck with it.

In order to get a remote session to each one, only one can have the RDP or SSH port at any time for the Service. 

So the setup resembles this:

• brianEhApp.cloudapp.net:3389 -> VM1:3389
• brianEhApp.cloudapp.net:56789 -> VM2:3389
• brianEhApp.cloudapp.net:54321 -> VM3:3389

This is relatively straightforward.  However, I ran into a situation what I was not able to authenticate to the higher ‘ephemeral’ ports that Azure was automatically generating as the administrative endpoint for my +1 machines.

So, I could use the Portal to change one VM then back out to the other and so on.

However, this was taking quite a long time.  And I am impatient.

So, I turned over to the Azure PowerShell cmdlets and whipped out the following script.  It switches the public endpoints for me and launches my administrative RDP session straight away.  Considerably faster than messing with the Portal.

$myVm = Read-Host -Prompt "what is the VM name you want to access via RDP?"
$myService = Read-Host -Prompt "what is the Service name where the VM resides?"
"The assumption is that the endpoint name is RDP"

# Check all the endpoints of all the VMs and move off 3389 to set desired VM to 3389
$deploymentVms = @()
$deploymentVms = Get-AzureVM -ServiceName $myService 

# First all VMs in the Service need to be off 3389 or there will be an error setting the correct one
foreach ($vm in $deploymentVms){
    $vmEnds = @()
    $vmEnds = Get-AzureEndpoint -VM $vm
    Foreach ($end in $vmEnds){
        If ($end.Port -eq 3389 -and $vm.Name -notmatch $myVm){
            $pubPort = Get-Random -Minimum 49152 -Maximum 65500
            Set-AzureEndpoint -Protocol tcp -LocalPort 3389 -PublicPort $pubPort -Name $end.Name -VM $vm | Update-AzureVM
        }
    }

}

# Now, set the desired VM Public endpoint to 3389
Get-AzureVM -ServiceName $myService -Name $myVm | Set-AzureEndpoint -Protocol tcp -LocalPort 3389 -PublicPort 3389 -Name RDP | Update-AzureVM

Get-AzureRemoteDesktopFile -ServiceName $myService -Name $myVm -Launch

Azure Virtual Machine the PowerShell Basics

Really, I am not going to go into a lot of explanation here.  I am assuming that you have some PowerShell background and need a bit to get started.  That is what I am covering.

You can find the cmdlets here:

https://www.windowsazure.com/en-us/manage/downloads/

Be sure to have your Azure management certificate properly stored in your Personal certificate store prior to connecting to your subscription.

These first commands are pretty much mandatory when you begin a PowerShell session.

Import the module:

​import-module 'C:\Program Files (x86)\Microsoft SDKs\Windows Azure\PowerShell\Azure\Azure.psd1'

Import a settings file (this speeds up as it lists all subscriptions you have access to - to create this file perform Export-AzurePublishSettingsFile (Visual Studio also uses this))

Import-AzurePublishSettingsFile 'C:\Users\Public\Documents\BrianEhServices.publishsettings'

Choose the subscription that you will interact with for your session:

Select-AzureSubscription -SubscriptionName "Sample Subscription"

Set the default Storage account that will be used (it must be in the same subscription)

Set-AzureSubscription -SubscriptionName "Sample Subscription" –CurrentStorageAccount SampleStorageAccount

By the way, do the above is handy.  Like me I assume that you all have alt least two Azure subscriptions (yours and the one your company has given you access to).  Using that settings file allows easy switching.  Now that you have that, go exploring.

If you have a Service or have deployed a Virtual Machine from the Gallery using the Portal you can query it, change it, dispose of it.

If you have VHD images you can manipulate those.  Now, disclaimer here, my experience so far is that CSUPLOAD from the Azure SDK is still the best way to get VHDs into Azure storage and it now supports both stateless and persistent VHD images.  It differentiates between the two because it registers them with the VHD repository for you and sets its life in motion.

But, I assume that you are getting itchy, so lets just begin with making a new Virtual Machine in a new Service.  The very same thing that you would get if you used a Gallery VM image (this is not the Quick VM as that would be New-AzureQuickVM).

First we need to find an image:

List all available images: Get-AzureVMImage

List all available in a table: Get-AzureVMImage | Format-Table

Find images that have been uploaded to your Storage account ('user' images): Get-AzureVMImage | where { ($_.Category -eq "user") }

Now I just want to use the Server 2012 Gallery image and create a VM (we will build on this command).

$svr2012Image = Get-AzureVMImage | where { ($_.Category -eq "Microsoft") -and ($_.Label -match "Server 2012" ) -and ($_.ImageName -match "Datacenter") }

Apply a customization configuration to the image:

$myImage = New-AzureVMConfig -Name MyNewMachine -InstanceSize ExtraSmall -ImageName $svr2012Image.ImageName
Add-AzureProvisioningConfig -VM $myImage -Windows –Password <complex password>

Create the Service and Provision the VM:

New-AzureVM -ServiceName "MyNewService" –VMs $myImage

Notice after it is finished that Azure automatically created an RDP endpoint to allow remote OS access (if it was a Linux image an SSH endpoint would have been created).

Azure IaaS Virtual Machine concepts

The Azure IaaS Virtual Machine is in Beta and anyone can request access.

For those have worked with Azure VMs in the past (Web, Worker, and VM Role) this is simply an addition, a new option.

There are some interesting contrasts, parallels, and dependencies when we consider both the PaaS and IaaS options (MSFT likes to draw the contrast that way, but fundamentally they are still machines what you can do and need to know is different).

MSFT already has lots of documents out there on the IaaS stuff.  I don’t intend to regurgitate what they have already documented.  They stuff they have not documented is far more interesting.

And there is a gallery of OS images that you can build machine from using the GUI portal.  And you can create and upload your own VHD images (a bit like VM Role).

In my next post I will begin exploring the PowerShell that is available to drive this stuff from the ground, that is a bit more interesting (at least to me).

First of all, what is the Virtual Machine?  If it isn’t a Web, Worker, or VM Role; what is it.

It is a bit like a VM Role, but without any agents to interact with the Azure Fabric, and the storage supports persistence.  Beyond persistence, it is just a machine.  Mind you, the way I state this it does not sound like much, but it took a lot for MSFT to work through extrapolating the PaaS provisioning and storage into supporting this.  There is a lot there.  Just go look up the TechEd talks.

Now, we all know that MSFT al about platforms, so the Virtual Machine by itself is just as uninteresting as it sounds.  Add in Virtual Networks and it gets a bit more interesting.

Virtual Networks allow defining of address allocation spaces.  And those can be divided into subnets.  If you know the SCVMM IP Pool model, it is similar to that with some implementation differences.

A Virtual Network is also a container or a boundary.  All machines within a Virtual Network can freely communicate with each other over the network. No matter the IP that was allocated to the VM, no IP subnet isolation.  A Virtual Network can also span services – provided they are in the same Affinity group and Region.  It all links.

A Virtual Network also supports the Gateway.  The Gateway is a way to ling the Virtual Network defined in Azure with an SSL VPN endpoint on the ground, in your enterprise.  This is just like any point-to-point VPN that you have used to connect branch offices, or to another business.  Before this we only had Azure Connect and its support for IPv6 and that it only acted as a machine to machine forwarder, fundamentally different.

One other concept is the VHD.  There are two types.  Image and OS. 

An Image can be provisioned into multiple VMs and is prepared so that Azure can customize it.  The Windows OS is prepared with sysprep, and the Linux OS is prepared with the Azure Linux Agent deprovision command.  The concept is the same.

When a new VM is created from an image it can and will be customized.  And, the Fabric also automatically creates a management endpoint (SSH or RDP).

An OS can be used by one Virtual Machine.  It is not provisioned, and therefore not customized.  It is just created an powered on.  You have to define everything, but it allows you to bring your own, pre-configured machine into Azure (without building it in space).

In the next few posts we will get a bit more into this and specifically driving it all with PowerShell.

The last concept is the Services.  IaaS Virtual Machines still run within a Service just like PaaS VMs.  But the two shall not mix.

You can create a Service without a deployment, this is androgynous, lacking either PaaS or IaaS gender.  As soon as the deployment is defined the service becomes either IaaS (meaning that it can run Virtual Machines) or PaaS (meaning that it can run VMs (or Roles)).  Thus the entire Service becomes stateless or persistent.

The two can interact over the network, but not through Azure Fabric Services.

I think that in the next post I am going to dive straight into the PowerShell.

PowerShell v3 everywhere

If you have not caught wind of this yet, you can download and install the “Windows Management Framework 3.0” on your boxes that are not Server 2012 or Windows 8.

As in the past this is more than just PowerShell, it includes WMI and WinRM compatibility updates.  (just not a big BITS update like v2 did).

You can find it here: http://www.microsoft.com/en-us/download/details.aspx?id=34595

This is for Windows 7 SP1 and Server 2008 SP2 or Server 2008 R2 SP1.

And, don’t forget Update-Help after you install it.

But you might as well be on your way to managing your Server 2012 / Windows 8 infrastructure the PowerShell way.

If you need .Net 4 you can find that for Server over here: http://www.microsoft.com/en-us/download/details.aspx?id=17718

And for Server Core here: http://www.microsoft.com/en-us/download/details.aspx?id=22833

If you run an OS that is older.  I am sorry, MSFT does not look back.  You can be one version old, but XP / 2003 is way out. 

FreeBSD on Server 2012 Hyper-V

This has been far more painful that I had expected.  And it seems that this experience was wrapped up in a few things.

Recently, there was an announcement of FreeBSD Integration Components being contributed to the world courtesy of Citrix, NetApp, and Microsoft.  http://freebsdonhyper-v.github.com/

Now, I thought I would give it a go.  So I downloaded the amd64 ISO image for FreeBSD 8.2 (the supported kernel for the Integration Components) and like a good person who hacks at things I downloaded the latest 9.0 release for good measure (just to try, you have to try after all).

Well, a good day went by – installing VMs, trying to get GIT installed, discovering that networking was not working properly, then name resolution was not working properly.  Sorting all that out….

Here is the low down of the experience so far:

Tip 1: Install FreeBSD into a VM using a Legacy Network Adapter. 

DHCP will properly work during installation (check ifconfig before you reboot, you want to know the IP you received).  Upon reboot, it all seems broken.  ifconfig shows no IP, major bummer.  A bit of searching will scratch up some random tidbits to get things working again.  Here is my summary to get past that:

Tip 2:  Before reboot write down your DHCP address (drop to a console if installing 8.2 and use ifconfig; 9.0 will show you in the GUI-ish network setup)

Tip 3:  After reboot, fix the DHCP assignment:

1. manually apply the IP that you wrote down using ifconfig: 
   1. ifconfig de0 192.168.3.5 netmask 255.255.255.0 (IP and netmask)
   2. ifconfig de0 down
   3. ifconfig de0 up
2. Now that you are on the network, release and renew to get DHCP to properly assign all the settings (if you didn’t ‘down’ and ‘up’, this won’t work).
   1. dhclient de0   (you should notice that it gets the IP from your DHCP server and applies it)

That was an immediate and temporary quick fix that I found through searching, however on reboot it means little to me as I have to manually do the workaround again.

Tip 4:  Modify so that DHCP will always work on boot

This is the one step that seems counter intuitive to me.  I thought I set DHCP to be enabled during the installation, why don’t I get an IP address?  I finally uncovered some clues through a few days of searching and learning and refining searches.  (it is amazing how the new still works like the old..)

Edit /etc/rc.conf:

Comment the following with a "#":  ifconfig_de0

Add the following:  ifconfig_de0="SYNCDHCP media 100baseTX mediaopt full-duplex"

This definitely needs more resiliency as we being to think about moving this VM about.  Additional searching shows this is tied to the DEC 100Mbps driver in FreeBSD (“de” device).  The edit forces the device to full-duplex.  Without this edit there is a duplex mismatch between the driver in the VM and the port of the vSwitch.  Interesting details..

Tip 5:  Fix the DNS settings so remote name resolution actually works (if it is not, as mine wasn’t / didn’t)

So, if you follow the instructions from github to install the Integration Components, step 2 is to install git.  Well, I tried and tried and DNS would not resolve.  Looking at /etc/resolv.conf I see my DNS servers, but nothing was resolving.

edit /etc/resolv.conf and add the word ‘search’ before your DNS servers name (which were auto inserted) – if it isn’t there. 

search brianeh.local

If that does not work add a public name server.  nameserver 199.7.83.42

I also had the problem where DNS requests that were forwarded were not coming back to the VM.  (I only had this problem after I set the hostname equal to the FQDN.  If I removed the domain from the hostname and only had a machine name then DNS forwarding worked properly.  So, it is obviously some strange FreeBSD thing that I don’t know enough about.  I did find a note that you can only have three, in the list.

You can find a list of public name servers here:  http://root-servers.org/

Tip 6:  Reboot after installing git. 

For some reason git is installed ( just by “pkg_add –r git” ) but the fetching of packages fails.  All that I can assume is that dependent services are not running.

Is that it?

Golly, I hope so.  I still could not get the Beta Hyper-V integration components to properly compile and run.  I built the kernel, did this and that.  I am sure it will come along, as there are other folks that will want to use a synthetic NIC (it was never detected).

I really hope this helps others and the project.

Server 2012 Core DHCP Server scope quick setup

I frequently have a need to use small VMS to quickly prototype things.  And lately I have been all deep in Server 2012.

Just to give myself some extra pain I have also been using Server Core whenever I have no absolute need for a GUI (getting off that Windows graphical console point-and-click bandwagon).

My latest adventure is to quickly create a DHCP Server. 

Just to get this running is really rather painless in the end.  And figuring out the proper PowerShell commands was pretty straightforward as well.

Begin with a Server 2012 Core installation.

Make sure it has a manually assigned IP.  ( you can use SCVMM 2012 SP1 CTP1/2 – or Set-NetIPAddress ).

Add the DHCP role: 

Add-WindowsFeature DHCP

Create a scope: 

Add-DhcpServerv4Scope –StartRange 192.168.1.5 –EndRange 192.168.1.254 –SubnetMask 255.255.255.0 –Name “192.168.1.x” –State Active

Believe it or not.  That is it.  Just create a VM and test it.

Mind you, this is just a DHCP server, I did not do any Active Directory integration or extra DHCP custom scope configuration.  I figure that will come in due time.

Okay, lets do a couple settings, especially for Provisioning Server fans.

Set-DhcpServerv4OptionDefinition –OptionId 66 –DefaultValue 192.168.1.2

Set-DhcpServerv4OptionDefinition –OptionId 67 –DefaultValue ARDBP32.BIN

Lets suppose you want to change the IP range.  First you must know the ScopeId, as that is required by the Set-

Get-DhcpServerv4Scope

Set-DhcpServerv4Scope –ScopeId <> –StartRange 192.168.1.100 –EndRange 192.168.1.254

If you want the DHCP server to respond to both DHCP and BOOTP then you can change that option as well:

Set-DhcpServerv4Scope –ScopeId <> –Type Both

Handling Import-VM errors in Server 2012 Hyper-V

Import-VM.  A great cmdlet for Hyper-V.

Couple that being able to import a VM that was not exported and you have a golden feature.

And if all the stars align it works like a charm.  Also notice that in the GUI you import a path, here you import a configuration file.

But, what if you have problems to fix?  It isn’t as straightforward.  You have to fix the source VM to match your host.

Even though there is an example under Get-Help it isn’t easy to visually parse and is difficult to follow, so lets add some color and additional explanation.

 

I have a VM.  That VM was running on Hyper-V Server X and I export a copy and move it to Hyper-V Server Y.

These servers are not set up the same, they are in different environments.

I try Import-VM and get the following results:

Import-VM -Path 'C:\users\Public\VmPxe\Virtual Machines\339C3412-E83F-4EA7-9DE6-F9D388B196E2.XML'

Import-VM : Unable to import virtual machine due to configuration errors.
Please use Compare-VM to repair the virtual machine.
At line:1 char:1
+ Import-VM -Path 'C:\users\Public\VmPxe\Virtual
Machines\339C3412-E83F-4EA7-9DE6- ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~
    + CategoryInfo          : NotSpecified: (:) [Import-VM], VirtualizationOpe
   rationFailedException
    + FullyQualifiedErrorId : Microsoft.HyperV.PowerShell.Commands.ImportVMCom
   mand

Not that error are easy to understand in PowerShell and your mileage varies by the folks that wrote a set, but I at least know that something is wrong and I have another cmdlet to find out what.  Fair enough.

Compare-VM -Path 'C:\users\Public\VmPxe\Virtual Machines\339C3412-E83F-4EA7-9DE6-F9D388B196E2.XML'

VM                 : Microsoft.HyperV.PowerShell.VirtualMachine
OperationType      : ImportVirtualMachine
Destination        : BJELKS4
Path               : C:\users\Public\VmPxe\Virtual
                     Machines\339C3412-E83F-4EA7-9DE6-F9D388B196E2.XML
SnapshotPath       : C:\users\Public\VmPxe\Snapshots
VhdDestinationPath :
VhdSourcePath      :
Incompatibilities  : {33012, 33012}

Excellent, I have incompatibilities 33012, twice.  What is that all about???  I guess that I need to capture that report to see the errors as I can see that it is in an array ( the {} told me so ).

$compareResult = Compare-VM -Path 'C:\users\Public\VmPxe\Virtual Machines\339C3412-E83F-4EA7-9DE6-F9D388B196E2.XML'

$compareResult.Incompatibilities

Message                                     MessageId Source
-------                                     --------- ------
Could not find Ethernet...                      33012 Microsoft.HyperV.Power...
Could not find Ethernet...                      33012 Microsoft.HyperV.Power...

$compareResult.Incompatibilities[0].Message
Could not find Ethernet switch 'VMs'.

Okay, some useful information.  Now, I need to fix that.  But how?  I have a Message, MessageId, and a Source.  I bet the Source is a Source object that can be acted upon. <light bulb on>

Lets just try to disconnect the VM from the virtual switch, but how? 

If I look at the –Examples under Get-Help I scan through and see that I can take this $compareResult and act upon the .Source objects.  What I have declared as variable $compareResult MSFT refers to as a “Compatibility Report”.  But the important part is that it is more than a report.

That array of objects behind $compareResult.Incompatibilities are actually objects.  They are references to the objects that are having problems.  Even though it began as an XML file.

In my case these are the VMNetworkAdapter objects of the VM that I am attempting to import.  So I can’t search on the name of the VMNetworkAdapter and are forced to work through the list of objects.  At this time it is a kind of temporary VM that is in a state of limbo, it is declared but not committed.

Lets work through and fix the VM that I am attempting to import.

$compareResult.Incompatibilities[0].Source | Disconnect-VMNetworkAdapter

Okay, that Disconnected one of the VMNetworkAdapaters, but I want to connect the other one to the proper Virtual Switch.

$compareResult.Incompatibilities[1].Source | Connect-VMNetworkAdapter -SwitchName "Virtual Network0"

compare-VM -CompatibilityReport $compareResult

VM                 : Microsoft.HyperV.PowerShell.VirtualMachine
OperationType      : ImportVirtualMachine
Destination        : BJELKS4
Path               : C:\users\Public\VmPxe\Virtual
                     Machines\339C3412-E83F-4EA7-9DE6-F9D388B196E2.XML
SnapshotPath       : C:\users\Public\VmPxe\Snapshots
VhdDestinationPath :
VhdSourcePath      :
Incompatibilities  :

Okay, looking good, no errors.  No what?  How do I import this silly Compatibility Report that I just modified?

Well, the option is there.

Import-VM -CompatibilityReport $compareResult

Name  State CPUUsage(%) MemoryAssigned(M) Uptime   Status
----  ----- ----------- ----------------- ------   ------
VmPxe Off   0           0                 00:00:00 Operating normally

My only big comment on this is that I have to act upon each individual .Incompatibilities.Message in the array on a case by case basis.  That is a bit of a pain.  And it took a while.  Especially because you have to be a bit familiar with the available PowerShell cmdlets to know what you might be able to apply to any particular object.

Seems that I need a script to automatically manage the incompatibilities…

My Windows Network Virtualization demo script

After all this work with WNV I wrapped everything up with a lunch time training session covering the entire topic.

Quite honestly, it is a big concept to get across in 1 hour, wrapping the brain around the management IP, the PA addresses, the CA addresses and getting that all straight is the most confusing part – the first time someone has this tossed at them.

Once folks get those three items all sorted out it all seems to fall into place naturally.

The other idea is that fact that you are doing demonstrations and presentations as if you were a management layer.  Not using a management layer, but you ARE the management layer.  You have to do everything that you would expect a SCVMM / CloudStack / vCenter to normally be doing on your behalf.

Here are the assumptions for my demo:

Two 2012 Hyper-V Servers, four VMs.  The VMs are Red11, Red22, Blue11, and Blue22.  Red and Blue represent different Customers / Tenants.  They are bringing their systems to my infrastructure and I will be hosting them on my two Hyper-V Servers.

Setup:

The VMs should have their IPs manually set in the following way and have PING enabled in the firewall.

• Red11 = 192.168.1.11, mask 255.255.255.0, gateway 192.168.1.1
• Red22 = 192.168.1.22, mask 255.255.255.0, gateway 192.168.1.1
• Blue11 = 192.168.1.11, mask 255.255.255.0, gateway 192.168.1.1
• Blue22 = 192.168.1.22, mask 255.255.255.0, gateway 192.168.1.1

The Hyper-V Servers should have one External Virtual network, with the VMs attached.  The Hyper-V Servers need to be joined to the same domain with Migration enabled (we will be moving one VM).  All VMs should be created on one of the Hyper-V Servers, this is also the Hyper-V Server where you will run the script.

And, the last point.  When you prepare the demo to run, be sure to power on all VMs first, but power on the Red VMs first, then the Blue VMs.  Red was on-boarded into my infrastructure first, and Blue’s VMs will catch an IP conflict.

Start-VM Red* -AsJob
sleep 60
Start-VM Blue* –AsJob

Once they are all running and report an IP address to the host, we are ready to go.

$vmNics = Get-VMNetworkAdapter *
do {
    $upCount = ($vmNics | where {$_.IPAddresses.count -ne 0})
    $upCount.Count
    sleep 30
}
until ($vmNics.Count -eq $upCount.Count)

At this time you can begin running through the following scenario, one command set at a time.  I used the PowerShell ISE and selected a block and then ran that block (F8).  If you want to make all of these commands into one-liners then you can use Start-Demo.ps1.

Here is the meat, with all my commentary in the comments.  Taking all the previous 6 posts and rolling them into something to demonstrate being the management layer.

<#
.SYNOPSIS
   This is the scripted demonstration. Of Windows Network Virtualization
.DESCRIPTION
   This assumes that there are two Server 2012 Hyper-V machines, joined to the
   same domain.  Both hypervisors have one External Virtual Switch.
   This script is executed on one hypervisor, and has portions against the
   remote hypervisor.
   That local hypervisor has 4 (Server 2012) VMs named Red11, Red22, Blue11, Blue22.
   On the remote hypervisor I used a PowerShell dedicated endpoint created
   to work around CredSSP issues - http://blogs.msdn.com/b/taylorb/archive/2012/03/26/remote-administration-with-powershell-3-0-sessions-part-1.aspx
.EXAMPLE
   For the demonstration the PowerShell v3 ISE was used to execute individual blocks of
   commands.
.AUTHOR
   Brian Ehlert
   Senior Software Test Engineer 2
   Citrix Labs, Redmond
#>

# Open a session to my remote host that I will simply re-use
$waldorf = New-PSSession waldorf.brianeh.local -ConfigurationName HVRemoteAdmin
Clear-Host

# This is just a demo, no fancy stuff.  Everything is hard coded.
# Fancy scripting is for other examples.

# Red and Blue are customers that want me to host their VMs.

# There are four VMs Red1 Red2 Blue1 Blue2. (they need to live on one host and get along)

# They happen to have overlapping IP address schemes of 192.168.1.x

# IP Conflict - second powered up / set will not work. The OS auto disables and APIPA
Get-VMNetworkAdapter * | Format-Table VMName, Name, MACAddress, VirtualSubnetID, IPAddresses -AutoSize

# I could open a ticket with the Network folks to build a VLAN. This will take too long.

# I could force one of the clients to reconfigure their IP scheme. This is too complicated.

# Instead, I have set unique VirtualSubnetIDs for each customer (isolation boundary)

Get-VMNetworkAdapter Red* | Set-VMNetworkAdapter -VirtualSubnetId 445566
Get-VMNetworkAdapter Blue* | Set-VMNetworkAdapter -VirtualSubnetId 7788990

Get-VMNetworkAdapter * | Format-Table VMName, Name, MACAddress, VirtualSubnetID, IPAddresses -AutoSize

# IP Conflict is resolved.

# To clear the APIPA status and the default behavior of taking the conflicting IP offline:
Get-VMNetworkAdapter -VMName Blue* | Disconnect-VMNetworkAdapter
Get-VMNetworkAdapter -VMName Blue* | Connect-VMNetworkAdapter -SwitchName VMs

Get-VMNetworkAdapter * | Format-Table VMName, Name, MACAddress, VirtualSubnetID, IPAddresses -AutoSize

# Start a ping from Red11 to Red22
vmconnect Statler.brianeh.local Red11

# This works until I have to distribute VMs because one host is not enough capacity to carry everything for both customers.

Move-VM -Name “Red22” -DestinationHost Waldorf.brianeh.local –IncludeStorage –DestinationStoragePath D:\Red22

# ping is now broken as there is no way to route this additional tag

# So I set up Windows Network Virtualization

# Enable the WNV binding
# At my local host
$vSwitch = Get-VMSwitch -SwitchType External
Enable-NetAdapterBinding -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription -ComponentID "ms_netwnv"

# At my Remote host
Invoke-Command -Session $waldorf {
$vSwitch = Get-VMSwitch -SwitchType External
Enable-NetAdapterBinding -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription -ComponentID "ms_netwnv"
}

# Define the Provider Address in my address space
# At my local host
$vSwitch = Get-VMSwitch -SwitchType External
$swPhysIf = Get-NetAdapter -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription
New-NetVirtualizationProviderAddress -InterfaceIndex $swPhysIf.InterfaceIndex -ProviderAddress 172.16.0.1 -PrefixLength 21

# At my Remote host
Invoke-Command -Session $waldorf {
$vSwitch = Get-VMSwitch -SwitchType External
$swPhysIf = Get-NetAdapter -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription
New-NetVirtualizationProviderAddress -InterfaceIndex $swPhysIf.InterfaceIndex -ProviderAddress 172.16.0.2 -PrefixLength 21
}

# Generate a GUID for each customer
$redGUID = [system.guid]::newguid()
$blueGUID = [system.guid]::newguid()

# Format the GUID string properly
$redGUID = "{" + [string]$redGUID + "}"
$blueGUID = "{" + [string]$blueGUID + "}"

# Define a Customer Route for each customer
# A the local host
New-NetVirtualizationCustomerRoute -RoutingDomainID $redGUID -VirtualSubnetID 445566 -DestinationPrefix "192.168.1.0/24“ -NextHop 0.0.0.0 -Metric 255
New-NetVirtualizationCustomerRoute -RoutingDomainID $blueGUID -VirtualSubnetID 7788990 -DestinationPrefix "192.168.1.0/24“ -NextHop 0.0.0.0 -Metric 255

# At the remote host
Invoke-Command -Session $waldorf -ArgumentList $redGUID, $blueGUID {
Param($redGUID, $blueGUID)
New-NetVirtualizationCustomerRoute -RoutingDomainID $redGUID -VirtualSubnetID 445566 -DestinationPrefix "192.168.1.0/24“ -NextHop 0.0.0.0 -Metric 255
New-NetVirtualizationCustomerRoute -RoutingDomainID $blueGUID -VirtualSubnetID 7788990 -DestinationPrefix "192.168.1.0/24“ -NextHop 0.0.0.0 -Metric 255
}

# Define the Lookup Routes on all switches that require it
# First local
New-NetVirtualizationLookupRecord -VMName Red11 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.11 -MACAddress 00155D002009 -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $redGUID
New-NetVirtualizationLookupRecord -VMName Red22 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.22 -MACAddress 00155D002008 -ProviderAddress 172.16.0.2 -Rule TranslationMethodEncap -CustomerID $redGUID
New-NetVirtualizationLookupRecord -VMName Blue11 -VirtualSubnetID 7788990 -CustomerAddress 192.168.1.11 -MACAddress 00155D00200B -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $blueGUID
New-NetVirtualizationLookupRecord -VMName Blue22 -VirtualSubnetID 7788990 -CustomerAddress 192.168.1.22 -MACAddress 00155D00200A -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $blueGUID

# Then remote
Invoke-Command -Session $waldorf -ArgumentList $redGUID, $blueGUID {
Param($redGUID, $blueGUID)
New-NetVirtualizationLookupRecord -VMName Red11 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.11 -MACAddress 00155D002009 -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $redGUID
New-NetVirtualizationLookupRecord -VMName Red22 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.22 -MACAddress 00155D002008 -ProviderAddress 172.16.0.2 -Rule TranslationMethodEncap -CustomerID $redGUID
New-NetVirtualizationLookupRecord -VMName Blue11 -VirtualSubnetID 7788990 -CustomerAddress 192.168.1.11 -MACAddress 00155D00200B -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $blueGUID
New-NetVirtualizationLookupRecord -VMName Blue22 -VirtualSubnetID 7788990 -CustomerAddress 192.168.1.22 -MACAddress 00155D00200A -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $blueGUID
}

# Ping is working again

# Take a look and the LookupRoutes
Get-NetVirtualizationLookupRecord | ft CustomerAddress, CustomerID, Provideraddress, macaddress, rule, vmname -AutoSize

# Break one of the routes, this can be due to a VM HA event, or a mis-configuraiton.
Remove-NetVirtualizationLookupRecord -CustomerAddress 192.168.1.22 -VMName Red22
New-NetVirtualizationLookupRecord -VMName Red22 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.22 -MACAddress 00155D002008 -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $redGUID

# Now the route is incorrect, what happens? (check ping)

# Repair the route

Remove-NetVirtualizationLookupRecord -CustomerAddress 192.168.1.22 -VMName Red22
New-NetVirtualizationLookupRecord -VMName Red22 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.22 -MACAddress 00155D002008 -ProviderAddress 172.16.0.2 -Rule TranslationMethodEncap -CustomerID $redGUID

Server 2012 Windows Network Virtualization part 6 Lookup Routes

I am finally at the last part that makes this all work.

If you have defined everything so far you have VM traffic routing between VMs on the same Virtual Switch (and it is isolated like a tiny VLAN) but nothing more.  No traffic will travel from VM to VM across hosts until we define the Lookup Route that tells that packet where to go.

here is the command in a nutshell:

New-NetVirtualizationLookupRecord -VirtualSubnetID 5001 -CustomerAddress 192.168.104.10 -MACAddress 00155D002001 -ProviderAddress 10.235.0.31 -Rule TranslationMethodNat -CustomerID "{2ad17590-33b5-45fc-ad3a-90e5ff9b017f}" -VMName VMPXE

The VirtualSubnetID, the CA (the IP of the VM), the MAC of the VM, the PA that the CA is associated with, the type of network virtualization to use, the Routing Domain (Customer ID) and the name of the VM (really for management benefits – it does not have to match anything).

The –Rules are TranslationMethodNat (for IP Rewrite) or TranslationMethodEncap for (NVGRE encapsulation).  This is that decision I mentioned before.

If your traffic is all IP based and can handle segmentation then by all means use encapsulation (also, your physical devises must support it, most do, but as long as they don’t somehow do something with the virtual subnet ID you should be good).

Otherwise fall back to Nat (IP Rewrite).  But, remember that if you use this you need one PA for each CA, so you need to add PA addresses each time you add a VM.  And don’t worry, you can define multiples on the physical NIC of the virtual switch.

Here I have a script to set up encapsulation on a static network (where I don’t have to worry about VM change).

Here are my assumptions:

1. The PA is defined and I am using one for all VMs
2. The VMs are running and reporting an IP address to Hyper-V
3. I write an output file to run on the ‘other’ Hyper-V Servers to propagate the Lookup Routes.
4. If the IP is 0.0.0.0 I ignore it.

There is a lot going on here and the screen does not format it well, be warned.

$paAddress = Get-NetVirtualizationProviderAddress

$otherHostsFile = $env:Public + "\Documents\" + "RunOnHostsOtherThan_" + $env:COMPUTERNAME + ".ps1"
Out-File -Force $otherHostsFile  # this is the propagation file

$vms = Get-VM -Name *

foreach ($i in $vms) {
    $i.VMName

    if ($i.NetworkAdapters.IPAddresses -ne $null) {
        $vmIpv4 = $i.NetworkAdapters.IPAddresses | where { $_.contains(".") }
        $vmIpv6 = $i.NetworkAdapters.IPAddresses | where { $_.contains(":") }
    }
    $vmNic = Get-VMNetworkAdapter -VM $i

    "    " + $vmNic.Name + " -> " + $vmNic.SwitchName

    if ($vmIpv4 -ne $NULL) {
        New-NetVirtualizationLookupRecord -VirtualSubnetID $vSubnetId -CustomerAddress $vmIpv4 -MACAddress $vmNic.MACAddress -ProviderAddress $paAddress.ProviderAddress -Rule TranslationMethodEncap -CustomerID $routingDomainId -VMName $vmNic.VMName
        Set-VMNetworkAdapter -VirtualSubnetId $vSubnetId -VMNetworkAdapter $vmNic
        ("New-NetVirtualizationLookupRecord -VirtualSubnetID " + $vSubnetId + " -CustomerAddress " + $vmIpv4 + " -MACAddress " + $vmNic.MACAddress + " -ProviderAddress " + $paAddress.ProviderAddress+ " -Rule TranslationMethodEncap -CustomerID `"" + $routingDomainId + "`" -VMName " + $vmNic.VMName) | Out-File -NoClobber -Append -FilePath $otherHostsFile

    }
    <#    If ($vmIpv6 -ne $NULL) {
        New-NetVirtualizationLookupRecord -VirtualSubnetID $vSubnetId -CustomerAddress $vmIpv6 -MACAddress $vmNic.MACAddress -ProviderAddress $paAddress.ProviderAddress -Rule TranslationMethodEncap -CustomerID $routingDomainId -VMName $vmNic.VMName
        Set-VMNetworkAdapter -VirtualSubnetId $vSubnetId -VMNetworkAdapter $vmNic
        ("New-NetVirtualizationLookupRecord -VirtualSubnetID " + $vSubnetId + " -CustomerAddress " + $vmIpv6+ " -MACAddress " + $vmNic.MACAddress + " -ProviderAddress " + $paAddress.ProviderAddress+ " -Rule TranslationMethodEncap -CustomerID `"" + $routingDomainId + "`" -VMName " + $vmNic.VMName) | Out-File -NoClobber -Append -FilePath $otherHostsFile
    }#>

    Clear-Variable -name vmIpv4 -ErrorAction SilentlyContinue
    Clear-Variable -Name vmIpv6 -ErrorAction SilentlyContinue
}

 

I have another script to set up NAT and that takes a bit more due to the PA management.  A post of its own.

Server 2012 Windows Network Virtualization part 5 the Customer Address

I am not going to spend much if any time on this.

Do you recall this graphic:

 

All that you need to know is that the Customer Address must be a known thing.  You need to know the IP address of your VM, the MAC or your VM, and how you are going to forward your VM traffic (encapsulation or IP Rewrite) prior to the next step.

Remember, if you use IP Rewrite – each CA needs a unique PA.  If you use encapsulation you can have one PA support multiple CAs.

IP Rewrite gives high compatibility but is more to manage, Encapsulation is easier to manage but might not work in all cases (actually, I know cases where it fails).

The CA is the IP of the VM.  There can only be one LookupRoute defined per CA + VirtualSubnetID.  Remember, VirtualSubnetID is your isolation boundary, within the CustomerRoute, but cannot be used more than once within a datacenter.

This can also be thought of as the IP that defines the target route – as a remote CA is actually the target of any packet.

And, most important – in the WNV model, this is the IP that the customer brings with them.  There can be duplicates, but not for a single customer – as that would duplicate too many settings and create conflicting routing rules.

Server 2012 Windows Network Virtualization part 4 the PA address

In the previous post I set the basics.  I enabled WNV and left you hanging.

Lets begin with getting the concepts straight about the PA (ProviderAddress) and the CA (CustomerAddress).  If you think about the paradigm, these names reflect the owner of the respective subnet / address space.

The PA is an address that is owned by the Provider.  Also referred to as the Hoster or the Enterprise.  These addresses allow packets to be routed between Hyper-V Virtual Switches.

The CA is an address that is brought by the Customer.  Also referred to as a Tenant, or a cloud user.  These addresses are the addresses that the VMs have.  And need to be known when establishing the routing rules.

Lets just focus on the PA for the remainder of this post.

An important thing to understand (that I really did not get at first) is that the ProviderAddress (the PA) is unique and different from the Management IP of the Hyper-V Server.

I draw it this way:

Also, the ProviderAddress just needs to be routable to any other hosts where VMs reside that participate in a particular virtual network.

For example:

Your management subnet might be 10.10.3.x.  If you don’t need any more than 254 PA addresses then you can use 192.168.3.x (for example).

In this model 192.168 is not routable, so all the Hyper-V Extensible Switches need to be local to each other. 

I am going to talk Switches here, since that is what this is all about, not the Hyper-V Server – that is a different IP and different issue.

ProviderAddresses are all about switch to switch communication using a routable IP address.  So all the networking or routing rules that you already know apply.  Your choice of PA addresses combined with routing rules can restrict your virtual network to a datacenter, or allow it to span datacenters, or even to remote services.  It is all in how you define and build it.

Lets add that you can define multiple PAs.  And the PAs can be private, public, different subnets, etc.  The IP of the PA simply interacts with the routing rules at the physical / fabric layer of your network.

Now, lets mention the two primary types of virtual networks.  MSFT has used the terms NVGRE (or Encapsulated) and IPRewrite (or NAT) in various places.  The reason that I mention this is that it affects your PA options.

IPRewrite sets up a NAT rule.  This requires one PA per CA.  So, each VM has an IP address (the CA) and a corresponding PA.  The rules for IPRewrite is essentially a one to one NAT (no port redirection here like in a router this is IP translation).

NVGRE uses encapsulation.  This is the most flexible in how you define it.  As you can use one PA for multiple CAs.  And again, you can have multiple PAs.  Here are some possibilities.

The simplest model of one PA per virtual switch:

This is compatible with:  NVRGE (all virtual networks using NVGRE)

 

Another possibility is one PA per virtual network.  This would give a bit more flexibility in managing different tenants sharing Hyper-V Servers. Again, this would only support using NVGRE (encapsulation).

 

If you want to use IPRewrite you need one PA for each VM.

After all this, you want to define a ProviderAddress?

It is really simple.  In this example I am assuming one thing, you have only one External Virtual Switch.

$wnvIf = Get-NetAdapterBinding -ComponentID "ms_netwnv" | Where-Object { $_.Enabled -eq $true } | Get-netadapter

New-NetVirtualizationProviderAddress -InterfaceIndex $wnvIf.InterfaceIndex -ProviderAddress 192.168.100.3 -PrefixLength 24

Next, we will tackle the CA space.