Hyper-V Resource Pool Introduction

There is a feature of Hyper-V 2012 that is rarely discussed but is highly useful.

First, the Resource Pool concept; 

For this I pulled a definition from the Office documentation; “A set of resources that is available for assignment”  that is the best way that I can describe it.

Second, this is not VMware style resource pools.  Their implementation is very unique.  It is closer to the XenServer implementation of resource pools.  However, it does not follow that either.

If you head over to the DMTF and search on Resource Pool you find something nice and vague about Resource Pool Hierarchies.  Okay, we will see that in the implementation.

So, before I move on, those of you with VMware backgrounds, just forget using resource pools to manage reservations and what not.  It might eventually get there, but not today.

As I mentioned in the Office quote, the Resource Pools represent assignment.  A connection.  A relationship (DMTF). 

If you want to see an easy example of the Resource Pools in action (and in the GUI), you need to create one.

The example

Lets look at Networking.  Connecting VMs, enabling VMs for all kinds of migrations, and inconsistent configurations.

I have two Hyper-V Servers.  They were set up by different folks, with different naming preferences.  Joe names his virtual network “VMs” and Gale names hers “LAN”.  Each time they move a VM back and forth they need to reconfigure the network settings of the VM. 

There has to be a way to do that without renaming their virtual switches.  There is.  Create an “Ethernet” Resource Pool.

Okay, now some PowerShell and some details

PS C:\Users\Administrator> get-command *resourcepool*

CommandType     Name                                               ModuleName
-----------     ----                                               ----------
Cmdlet          Get-VMResourcePool                                 Hyper-V
Cmdlet          Measure-VMResourcePool                             Hyper-V
Cmdlet          New-VMResourcePool                                 Hyper-V
Cmdlet          Remove-VMResourcePool                              Hyper-V
Cmdlet          Rename-VMResourcePool                              Hyper-V
Cmdlet          Set-VMResourcePool                                 Hyper-V

If you have done nothing with Resource Pools on a Hyper-V Server and you simply type Get-VMResourcePool you actually get a bunch back.

PS C:\Users\Administrator> Get-VMResourcePool

Name       ResourcePoolType       ParentName ResourceMeteringEnabled
----       ----------------       ---------- -----------------------
Primordial FibreChannelConnection            False
Primordial FibreChannelPort                  False
Primordial ISO                               False
Primordial VFD                               False
Primordial VHD                               False
Primordial Ethernet                          False
Primordial Memory                            False
Primordial Processor                         False

These are the Primordial Resource Pools.  Hyper-V gives these to you because you can only add children, the Server must provide the first Parent.  In this case they called it Primordial.  Sounds all medieval doesn’t it?  It simply means there is nothing before, it is definitely the Root, not some Pool that someone called “root”.

So, lets make a new Resource Pool and attach it to the Ethernet Primordial Pool.

PS C:\Users\Administrator> New-VMResourcePool -Name "VM LAN" -ResourcePoolType Ethernet

Name   ResourcePoolType ParentName   ResourceMeteringEnabled
----   ---------------- ----------   -----------------------
VM LAN Ethernet         {Primordial} False

Notice how the Primordial pool was assumed in this case?  Handy.

Now, I could have explicitly defined –ParentName and I probably would have if I had multiples.  Because I could branch Resource Pools if I wanted to. 

If I branch Resource Pools I can use them to create logical groupings for metering or connecting devices.  And each one would have different options because it is at a different level and combination of parents.

This familial stuff can get pretty messy so I will keep this example simple.

Now that I have my Resource Pool.  How do I use it?

Well, now that I created a Resource Pool. If I open the settings of the Network Adapter for any VM on that Hyper-V Server I see something totally new.

And if I drop the selection list I can connect to the Primordial ( “<Root>” ) or the Resource Pool I just created.

If I select the Pool I just created the Virtual Switch setting is changed.

Because I didn't associate a Virtual Switch with my new Resource Pool.  This command is not totally intuitive, I expect to use Set-VMSwitch to modify the setting, however, the clever PM behind the Hyper-V cmdlets decided to use a different verb.

PS C:\Users\Administrator> Add-VMSwitch -ResourcePoolName "VM LAN" -Name VMs

Now, If I open the settings of the VM again.  It makes sense to select “automatic connection” for the switch.  This way the Pool is connected to, not the switch.

I can actually name the switch in some way unique to the server or hardware, and have the consistent naming abstracted above that.  So, whatever switch is associated with this Pool, the VM will be connected to it.

What else can I do with that Resource Pool?

Well, I can Measure it of course.

First, enable Metering then measure it.

PS C:\Users\Administrator> Enable-VMResourceMetering -ResourcePoolType Ethernet -Name "VM LAN"
PS C:\Users\Administrator> Measure-VMResourcePool -ResourcePoolType Ethernet -Name "VM LAN"

Name   ResourcePoolType AvgCPU(MHz) AvgRAM(M) TotalDisk(M) NetworkInbound(M) NetworkOutbound(M)
----   ---------------- ----------- --------- ------------ ----------------- ------------------
VM LAN {Ethernet}                                          0                 0

Hmm, seems that no time passed, so there is no data.  Waiting a bit, lets Measure again.

PS C:\Users\Administrator> Measure-VMResourcePool -ResourcePoolType Ethernet -Name "VM LAN"

Name   ResourcePoolType AvgCPU(MHz) AvgRAM(M) TotalDisk(M) NetworkInbound(M) NetworkOutbound(M)
----   ---------------- ----------- --------- ------------ ----------------- ------------------
VM LAN {Ethernet}                                          2                 1

Hey, some network traffic.  Excellent.

This is just an introduction to Resource Pools.  I hope to bring some more in the future as they are highly useful, yet relatively invisible.

My Windows Network Virtualization demo setup script

In the past I posted my two Hyper-V Server demo script for Windows Network Virtualization.

And a whole series of posts about Windows Network Virtualization.  Unfortunately, some folks just could not get it working.

Well, a key script that I left out was the one I used to setup my servers.  And that is here.

I use the little trick of setting the IP of the VMs through WMI as well (remember, the VMs must be running).

I also use a little trick to build all of the routes and rules.  Remember, there is no automatic route maintenance, if you are setting this up and using it, you must also manage the change – the system won’t do it for you.

One requirement here, the machines are joined to the same domain.  And I have a Register-PSSessionConfiguration on the remote machine defined as Taylor mentions here.

Okay, enough of that.  Here is my setup and reset PowerShell script:

 

$what = Read-Host -Prompt "Do you want to RESET or SETUP?"

# Load the function to set the IP address.
function Get-VMIPAddress ($VMName)
{
    $Query = "SELECT * FROM Msvm_ComputerSystem WHERE ElementName = '" + $VMName + "'"
    $VM = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    if ($VM -eq $null)
    {
        write-host ("No VM found with the specified name '" + $VMName + "'")
        return
    }

    $Query = "ASSOCIATORS OF {" + $VM.__PATH + "} WHERE ResultClass = Msvm_VirtualSystemSettingData"
    $VMSettings = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    $Query = "ASSOCIATORS OF {" + $VMSettings.__PATH + "} WHERE ResultClass = Msvm_SyntheticEthernetPortSettingData"
    $VMNIC = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    $Query = "ASSOCIATORS OF {" + $VMNIC.__PATH + "} WHERE ResultClass = Msvm_GuestNetworkAdapterConfiguration"
    $VMIpSettings = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    $VMIpSettings
}

function Set-VMIPAddress ($VMName, $IPSettings)
{
    $Service = Get-WmiObject -Class "Msvm_VirtualSystemManagementService" -Namespace "root\virtualization\v2"

    $Query = "SELECT * FROM Msvm_ComputerSystem WHERE ElementName = '" + $VMName + "'"
    $VM = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    if ($VM -eq $null)
    {
        write-host ("No VM found with the specified name '" + $VMName + "'")
        return
    }

    $Value = $Service.SetGuestNetworkAdapterConfiguration($VM, $IPSettings.GetText(1))

    #Value success if the return value is "0"
    if ($Value.ReturnValue -eq 0)
    {
        write-host "Operation completed successfully"
    }

    #If the return value is not "0" or "4096" then the operation failed
    elseif ($Value.ReturnValue -ne 4096)
    {
        write-host ("Operation failed: " + $Value.ReturnValue)
    }

    else
    {
        #Get the job object
        $job = [WMI]$Value.job

        if ($job.JobState -eq 7)
        {
            # Job has already failed

            write-host "Operation failed"
            write-host "ErrorCode:" $job.ErrorCode
            write-host "ErrorDescription" $job.ErrorDescription
        }
        else
        {
            #Provide updates if the jobstate is "3" (starting) or "4" (running)
            while ($job.JobState -eq 3 -or $job.JobState -eq 4)
            {
                write-host $job.PercentComplete
                start-sleep 1

                #Refresh the job object
                $job = [WMI]$Value.job

                #A jobstate of "7" means success
                if ($job.JobState -eq 7)
                {
                    write-host "Operation completed successfully"
                }
                else
                {
                    write-host "Operation failed"
                    write-host "ErrorCode:" $job.ErrorCode
                    write-host "ErrorDescription" $job.ErrorDescription
                }
            }
        }
    }
}

switch ($what) {
    RESET {
        "RESET!!"
        # Get set up and make sure that it is all ready

        Invoke-Command -ComputerName Waldorf.brianeh.local -ConfigurationName HVRemoteAdmin { Move-VM -Name “Red22” -DestinationHost Statler.brianeh.local –IncludeStorage –DestinationStoragePath D:\BlankVM }

        sleep 30

        # Reset adapters
        Get-VMNetworkAdapter -VMName * | Connect-VMNetworkAdapter -SwitchName VMs
        Get-VMNetworkAdapter -VMName * | Set-VMNetworkAdapter -VirtualSubnetId 0

        Get-VMNetworkAdapter * | ft vmname, name, macaddress, ipaddresses, virtualsubnetid

        # Reset the VMs to DHCP
        # Red11
        $VMIP = Get-VMIPAddress "Red11"
        $VMIP.DHCPEnabled = $True
        Set-VMIPAddress "Red11" $VMIP

        # Red22
        $VMIP = Get-VMIPAddress "Red22"
        $VMIP.DHCPEnabled = $True
        Set-VMIPAddress "Red22" $VMIP

        # Blue11
        $VMIP = Get-VMIPAddress "Blue11"
        $VMIP.DHCPEnabled = $True
        Set-VMIPAddress "Blue11" $VMIP

        # Blue22
        $VMIP = Get-VMIPAddress "Blue22"
        $VMIP.DHCPEnabled = $True
        Set-VMIPAddress "Blue22" $VMIP

        # Reset the Windows NetVirt binding

        $vSwitch = Get-VMSwitch -SwitchType External
        Disable-NetAdapterBinding -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription -ComponentID "ms_netwnv"

        Invoke-Command -ComputerName waldorf.brianeh.local -ConfigurationName HVRemoteAdmin {
            $vSwitch = Get-VMSwitch -SwitchType External
            Disable-NetAdapterBinding -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription -ComponentID "ms_netwnv"
        }

        # power off VMs, Red first
        Stop-VM Red* -AsJob
        sleep 60
        Stop-VM Blue* -AsJob

    }

    SETUP {
        "SETUP!!"
        # power on VMs, Red first
        Start-VM Red* -AsJob
        sleep 60
        Start-VM Blue* -AsJob

        # wait until all VMs ICs are 'awake' and are reporting an IP address
        $vmNics = Get-VMNetworkAdapter *
        do {
            $upCount = ($vmNics | where {$_.IPAddresses.count -ne 0})
            $upCount.Count
            sleep 30
        }
        until ($vmNics.Count -eq $upCount.Count)

        # Red11
        # Set the IP of each VM
        $VMIP = Get-VMIPAddress "Red11"

        #Setting the VM Network settings to a static IPv4 address
        $VMIP.DNSServers = @("192.168.1.2")
        $VMIP.IPAddresses = @("192.168.1.11")
        $VMIP.DefaultGateways = @("192.168.1.1")
        $VMIP.Subnets = @("255.255.255.0")
        $VMIP.DHCPEnabled = $False

        Set-VMIPAddress "Red11" $VMIP

        # Red22
        $VMIP = Get-VMIPAddress "Red22"

        $VMIP.DNSServers = @("192.168.1.2")
        $VMIP.IPAddresses = @("192.168.1.22")
        $VMIP.DefaultGateways = @("192.168.1.1")
        $VMIP.Subnets = @("255.255.255.0")
        $VMIP.DHCPEnabled = $False

        Set-VMIPAddress "Red22" $VMIP

        # Blue11
        $VMIP = Get-VMIPAddress "Blue11"

        $VMIP.DNSServers = @("192.168.1.2")
        $VMIP.IPAddresses = @("192.168.1.11")
        $VMIP.DefaultGateways = @("192.168.1.1")
        $VMIP.Subnets = @("255.255.255.0")
        $VMIP.DHCPEnabled = $False

        Set-VMIPAddress "Blue11" $VMIP

        # Blue22
        $VMIP = Get-VMIPAddress "Blue22"

        $VMIP.DNSServers = @("192.168.1.2")
        $VMIP.IPAddresses = @("192.168.1.22")
        $VMIP.DefaultGateways = @("192.168.1.1")
        $VMIP.Subnets = @("255.255.255.0")
        $VMIP.DHCPEnabled = $False

        Set-VMIPAddress "Blue22" $VMIP

        Get-VMNetworkAdapter * | ft vmname, name, macaddress, ipaddresses, virtualsubnetid

    }
}

Securing Azure Virtual Machines or I got hacked and how you shouldn’t

I have been working with Azure for over a year now and this is the first time I have had a machine compromised.  I will say that this entire incident has been a good learning experience.  I think that anyone can learn from my mistakes here.

Thinking back this event is interesting from many angles.

• a default machine provisioning behavior is opening a door that you (as the customer) needs to remember to close or not open in the first place.
• the machine is really out there, in the wild, innocent, vulnerable.  Not protected in your datacenter behind all the other protections.
• your IT Security folks might not yet be engaged in how to deal with deploying into public clouds.
• you are doing test, exploration, or other work on systems that are not going to production.

 

Let’s go back in time to the week of November 26. 

My innocent Virtual Machine was compromised by an attack against the RDP protocol.  The well known RDP port 3389 was the vector.

The attacker succeeded in obtaining access, changing the local administrator password and pwning my box.  I can only image that it was then ‘sold’ on some botnet as later it ended up with a PayPal phishing site.

Then it began the process of being capacity to begin finding other vulnerable folks. This is where I noticed what was happening.

(BTW – the MSFT folks that handle this type of incident were extremely helpful in working out what happened, in what order, and how.)

 

First, lets look at the environment where I put my machine.

It was in Azure.  Placing any machine in Azure is equivalent to placing it in the corporate DMZ.  There is a load balancer between you and the outside world, and public endpoint definition(s) that define the public ingress port(s).

Beyond that, you can actually do bunches of different configurations of Virtual Networks, or Gateways.  You can link Services.  All kinds of stuff.  But remember, all IaaS machine to machine network traffic within a Service is wide open.  There are no private endpoints (as in a PaaS Service).

That said, any IT Pro would immediately say ‘duh’ or ‘you idiot’.  Yep.  Nothing I can comment there.

 

There are some default IaaS behaviors that are enabling this;

• automatically opening an RDP endpoint for management
• requiring complex passwords (false sense of security)
• local administrator is not disabled
• no real guidance around practices (not that anyone RTFM’s any longer anyway – they simply SearchTFM)

PaaS Roles don’t have these behaviors.  And the RDP connection is secured with a Certificate and a secondary user account.

Some of the Azure MVPs found my original MSDN forum thread and beat my blog post out.  My IT Pro background is leading me to some different ideas, and I felt I needed to tell a bigger story.

And, as I think about this; remember that you are administering these lockdowns at the same time that your machine could be getting probed, or brute force attacked.  So, choose a logical order, define a strong password at the point to machine creation (however you do it).  At least it gives you a fighting chance.

 

So, this is where i mention a bit of guidance.  Here are some options to pick and choose from.

Some initial mitigation suggestions to choose from:

• Use the PaaS roles of Web or Worker as the front end machines whenever possible.  They are already hardened.
• Rename the local administrator account.
• Disable the local administrator account and create some uncommonly named user account for administrative access.
• Choose strong plus complex passwords, or passphrases.  Not simply one or the other.  The OS can enforce complexity but not strength. 
• A dictionary attack is likely to hit “P@ssw0rd” but it is unlikely to hit “Just a city boy, born and raised in South Detroit”.
• My favorite example of password strength: http://xkcd.com/936/ 
• Denying user access after X failed logon attempts (lock the account).  This is a Local security policy if not domain joined, or a Domain policy if joined.  Consider an automatic (timed) unlock as well, or you could have no recourse but to destroy your machine.
• Do not allow the creation of the default RDP public endpoint.  This is only possible through the API / PowerShell.  Or delete the auto created endpoint after creating the machine in the Portal.
• Only create the RDP endpoint when remote administration is necessary, and removing it after.  But remember that we are human, and unless you have some interface doing this for you, you will probably forget at some point.
• Remove the RDP endpoint and use the Virtual Network Gateway feature of the Azure Virtual Network for secured remote administration without public endpoints.  This requires some ground based router, and the VPN is slow, but your ports are closed.
• Remove RDP endpoint & use Azure Connect.  This is limited to IPv6 TCP traffic only, but that should cover anything required to manage the OS.
• Avoid 3389 as the public port (I noticed my compromised machine specifically scanning for this port to spread itself) by using a port in the ephemeral range.
• Use the Windows Advanced Firewall rules and define them appropriately.
• Use Windows IP Security Policies and tightly define the sources from which RDP traffic can be accepted from.  This is highly effective, but a pain to set up.
• Monitor the machine.  Azure provides metrics through the portal and API.  Discover a baseline.  Use an agent within the machine.  This only detects the compromise after it happens and is not preventative.
• Take a snapshot of the clean state.  This is not a point and click thing in Azure today, but you can work this out using the Storage cmdlets through destroying your machine, making the diff disk, and reincarnating the machine.

 

Some of these practices are simply security through obscurity but, in this case, I think it’s OK.

• Changing the name of the administrator account dramatically increases the search space for a username/password pair.
• Changing the RDP endpoint is going to increase the search space for a TCP port/username/password trio.

I didn’t mention antivirus.  I won’t go into my theories there.  My personal opinion is that if all the tightening measures are done and you still get hacked, most likely your AV would have failed as well.

Always be ready for the possibility of rebuilding a machine.  Never assume you can simply move forward, or even install a backup.  The best you can do is revert to a known clean state.

 

Good luck!

Setting VM IP with Hyper-V WMI

There is a hidden feature in Server 2012.  Buried deep in the bowels of the WMI classes is this little tidbit that is discoverable, but not talked about.

You can set the IP of a VM through Hyper-V 2012. 

For a long time now, in the forums we have told folks that this cannot be done.  Well, it can be.  And at small scale it works.  I have used it to set up my Network Virtualization demonstration environment.

The thing that you need to be aware of is that this is one of those cases where you send your command to WMI / CIM and you need to double back and check what happened.  Did the IP actually set? 

Be sure to check that before going off and expecting that it did.  If it didn’t, you have event logs in the VM.  Check there.

This took a bit of working through and discovery.  I never did work through IPv6, I stopped when I got IPv4 working.

<speculation>

As you might imagine, there has to be a dependency on the aligning of the version of the Integration Services within the VM and the Hyper-V Server.  I could not imagine this working with Hyper-V 2012 and a Server 2008 VM when the Integration Components in the VM have not been updated to the current Hyper-V level.

</speculation>

One thing that I did not blog about with the Network Virtualization script was how I set up my environment, more on that next, I scripted it, and it is not small.

Back the the subject, setting the IP of a VM through WMI of the Hyper-V Server.  I am going to leave out all the error handling just to make this easier to read through.

Here is a PowerShell function where you can see the WMI in action:

function Set-VMIPAddress ($VMName, $IPSettings)

{

    $Service = Get-WmiObject -Class "Msvm_VirtualSystemManagementService" -Namespace "root\virtualization\v2"

    $Query = "SELECT * FROM Msvm_ComputerSystem WHERE ElementName = '" + $VMName + "'"

    $VM = Get-WmiObject -Query $Query -Namespace "root\virtualization\v2"

    $setIP = $Service.SetGuestNetworkAdapterConfiguration($VM, $IPSettings.GetText(1))

}

Of course, you want to consider $setIP.ReturnValue

Like all WMI / CIM commands if the return value is “0” then you have success, “4096” generally means it is running.  And anything else means something went wrong in most cases.

If you have “4096” then query for the status of the job itself.  $job = [WMI]$setIP.job

By the way, if you want to get the IP of the VM don’t think $Service.GetGuestNetworkAdapterConfiguration.  This is proper thinking for PowerShell but not for WMI / CIM.  CIM could never be that easy.  ;-)

Look for the class Msvm_GuestNetworkAdapterConfiguration to get the IP of a VM.  Find the VM NIC with Msvm_SyntheticEthernetPortSettingData first.