WAP Gallery Image, Dynamic IP address, and the SCVMM DHCP switch extension

Recently I had to put together a hands on lab for a number of sales engineers.
The lab involved SCVMM Service Templates, a custom Windows Azure Pack Gallery Image, and a Desired State Configuration module.

I had my environment of Hyper-V 2012 R2, SCVMM 2012 R2, and WAP about 95% configured.  As much as I could and still support the students re-using my VMs with their own Hyper-V Server.

Since the lab was not about WAP, but instead about my gallery image, I wanted to keep it as simple as possible.  I had a cloud, the cloud had a VM Network assigned, the students created a static IP pool.

(I already had an Internal Virtual Switch being created by SCVMM as a Logical Switch so that all lines of dependency were properly drawn)

In the WAP Admin portal - I had the students add the cloud and the VM Network to their plan.

I deploy my Gallery Image, and the domain join failed. 
I look closer, and I see that my VM ended up with an APIPA address and not an address from the IP Pool. 

Come to find out, the default behavior of a WAP Gallery Image is for dynamic IP address assignment. 
Which, if you only ever deploy a gallery image to a Windows Network Virtualization VM Network, you will never notice.  You will instead see that you get an IP from the IP Pool.

Something that I discovered long ago was that there is a custom Hyper-V Virtual Switch extension that ships with SCVMM.  It is actually a DHCP responder.  It catches the IP request, notifies SCVMM, and SCVMM responds with an IP from the SCVMM IP Pool assigned to the VM.  Nifty.

But, this path only happens if the VM is attached to a Windows Network Virtualization (NVGRE) network managed by SCVMM.

Back to the default Gallery Image behavior of a dynamic IP address.  No WNV network, no IP from an IP Pool.  How to fix this?

The only way to fix this is to open the Resource Definition of the Gallery Image, and then open the Network Profile, then the NIC.
And change the AllocationMethod to Static.

While you are in there, you will most likely notice a number of other interesting settings.

But the thing to be aware of is this, these are hard coded values, unless you work through making them settings that are actually exposed to your end customer (at this time you can't expose these settings).
If you change a setting here, that makes a dependency on an SCVMM placement rule, SCVMM will have to find a place that this VM can go to support all of the settings.  If it cannot, your VM will not be deployed.  And your tenant will call.

My Windows Network Virtualization demo script

After all this work with WNV I wrapped everything up with a lunch time training session covering the entire topic.

Quite honestly, it is a big concept to get across in 1 hour, wrapping the brain around the management IP, the PA addresses, the CA addresses and getting that all straight is the most confusing part – the first time someone has this tossed at them.

Once folks get those three items all sorted out it all seems to fall into place naturally.

The other idea is that fact that you are doing demonstrations and presentations as if you were a management layer.  Not using a management layer, but you ARE the management layer.  You have to do everything that you would expect a SCVMM / CloudStack / vCenter to normally be doing on your behalf.

Here are the assumptions for my demo:

Two 2012 Hyper-V Servers, four VMs.  The VMs are Red11, Red22, Blue11, and Blue22.  Red and Blue represent different Customers / Tenants.  They are bringing their systems to my infrastructure and I will be hosting them on my two Hyper-V Servers.

Setup:

The VMs should have their IPs manually set in the following way and have PING enabled in the firewall.

• Red11 = 192.168.1.11, mask 255.255.255.0, gateway 192.168.1.1
• Red22 = 192.168.1.22, mask 255.255.255.0, gateway 192.168.1.1
• Blue11 = 192.168.1.11, mask 255.255.255.0, gateway 192.168.1.1
• Blue22 = 192.168.1.22, mask 255.255.255.0, gateway 192.168.1.1

The Hyper-V Servers should have one External Virtual network, with the VMs attached.  The Hyper-V Servers need to be joined to the same domain with Migration enabled (we will be moving one VM).  All VMs should be created on one of the Hyper-V Servers, this is also the Hyper-V Server where you will run the script.

And, the last point.  When you prepare the demo to run, be sure to power on all VMs first, but power on the Red VMs first, then the Blue VMs.  Red was on-boarded into my infrastructure first, and Blue’s VMs will catch an IP conflict.

Start-VM Red* -AsJob
sleep 60
Start-VM Blue* –AsJob

Once they are all running and report an IP address to the host, we are ready to go.

$vmNics = Get-VMNetworkAdapter *
do {
    $upCount = ($vmNics | where {$_.IPAddresses.count -ne 0})
    $upCount.Count
    sleep 30
}
until ($vmNics.Count -eq $upCount.Count)

At this time you can begin running through the following scenario, one command set at a time.  I used the PowerShell ISE and selected a block and then ran that block (F8).  If you want to make all of these commands into one-liners then you can use Start-Demo.ps1.

Here is the meat, with all my commentary in the comments.  Taking all the previous 6 posts and rolling them into something to demonstrate being the management layer.

<#
.SYNOPSIS
   This is the scripted demonstration. Of Windows Network Virtualization
.DESCRIPTION
   This assumes that there are two Server 2012 Hyper-V machines, joined to the
   same domain.  Both hypervisors have one External Virtual Switch.
   This script is executed on one hypervisor, and has portions against the
   remote hypervisor.
   That local hypervisor has 4 (Server 2012) VMs named Red11, Red22, Blue11, Blue22.
   On the remote hypervisor I used a PowerShell dedicated endpoint created
   to work around CredSSP issues - http://blogs.msdn.com/b/taylorb/archive/2012/03/26/remote-administration-with-powershell-3-0-sessions-part-1.aspx
.EXAMPLE
   For the demonstration the PowerShell v3 ISE was used to execute individual blocks of
   commands.
.AUTHOR
   Brian Ehlert
   Senior Software Test Engineer 2
   Citrix Labs, Redmond
#>

# Open a session to my remote host that I will simply re-use
$waldorf = New-PSSession waldorf.brianeh.local -ConfigurationName HVRemoteAdmin
Clear-Host

# This is just a demo, no fancy stuff.  Everything is hard coded.
# Fancy scripting is for other examples.

# Red and Blue are customers that want me to host their VMs.

# There are four VMs Red1 Red2 Blue1 Blue2. (they need to live on one host and get along)

# They happen to have overlapping IP address schemes of 192.168.1.x

# IP Conflict - second powered up / set will not work. The OS auto disables and APIPA
Get-VMNetworkAdapter * | Format-Table VMName, Name, MACAddress, VirtualSubnetID, IPAddresses -AutoSize

# I could open a ticket with the Network folks to build a VLAN. This will take too long.

# I could force one of the clients to reconfigure their IP scheme. This is too complicated.

# Instead, I have set unique VirtualSubnetIDs for each customer (isolation boundary)

Get-VMNetworkAdapter Red* | Set-VMNetworkAdapter -VirtualSubnetId 445566
Get-VMNetworkAdapter Blue* | Set-VMNetworkAdapter -VirtualSubnetId 7788990

Get-VMNetworkAdapter * | Format-Table VMName, Name, MACAddress, VirtualSubnetID, IPAddresses -AutoSize

# IP Conflict is resolved.

# To clear the APIPA status and the default behavior of taking the conflicting IP offline:
Get-VMNetworkAdapter -VMName Blue* | Disconnect-VMNetworkAdapter
Get-VMNetworkAdapter -VMName Blue* | Connect-VMNetworkAdapter -SwitchName VMs

Get-VMNetworkAdapter * | Format-Table VMName, Name, MACAddress, VirtualSubnetID, IPAddresses -AutoSize

# Start a ping from Red11 to Red22
vmconnect Statler.brianeh.local Red11

# This works until I have to distribute VMs because one host is not enough capacity to carry everything for both customers.

Move-VM -Name “Red22” -DestinationHost Waldorf.brianeh.local –IncludeStorage –DestinationStoragePath D:\Red22

# ping is now broken as there is no way to route this additional tag

# So I set up Windows Network Virtualization

# Enable the WNV binding
# At my local host
$vSwitch = Get-VMSwitch -SwitchType External
Enable-NetAdapterBinding -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription -ComponentID "ms_netwnv"

# At my Remote host
Invoke-Command -Session $waldorf {
$vSwitch = Get-VMSwitch -SwitchType External
Enable-NetAdapterBinding -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription -ComponentID "ms_netwnv"
}

# Define the Provider Address in my address space
# At my local host
$vSwitch = Get-VMSwitch -SwitchType External
$swPhysIf = Get-NetAdapter -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription
New-NetVirtualizationProviderAddress -InterfaceIndex $swPhysIf.InterfaceIndex -ProviderAddress 172.16.0.1 -PrefixLength 21

# At my Remote host
Invoke-Command -Session $waldorf {
$vSwitch = Get-VMSwitch -SwitchType External
$swPhysIf = Get-NetAdapter -InterfaceDescription $vSwitch.NetAdapterInterfaceDescription
New-NetVirtualizationProviderAddress -InterfaceIndex $swPhysIf.InterfaceIndex -ProviderAddress 172.16.0.2 -PrefixLength 21
}

# Generate a GUID for each customer
$redGUID = [system.guid]::newguid()
$blueGUID = [system.guid]::newguid()

# Format the GUID string properly
$redGUID = "{" + [string]$redGUID + "}"
$blueGUID = "{" + [string]$blueGUID + "}"

# Define a Customer Route for each customer
# A the local host
New-NetVirtualizationCustomerRoute -RoutingDomainID $redGUID -VirtualSubnetID 445566 -DestinationPrefix "192.168.1.0/24“ -NextHop 0.0.0.0 -Metric 255
New-NetVirtualizationCustomerRoute -RoutingDomainID $blueGUID -VirtualSubnetID 7788990 -DestinationPrefix "192.168.1.0/24“ -NextHop 0.0.0.0 -Metric 255

# At the remote host
Invoke-Command -Session $waldorf -ArgumentList $redGUID, $blueGUID {
Param($redGUID, $blueGUID)
New-NetVirtualizationCustomerRoute -RoutingDomainID $redGUID -VirtualSubnetID 445566 -DestinationPrefix "192.168.1.0/24“ -NextHop 0.0.0.0 -Metric 255
New-NetVirtualizationCustomerRoute -RoutingDomainID $blueGUID -VirtualSubnetID 7788990 -DestinationPrefix "192.168.1.0/24“ -NextHop 0.0.0.0 -Metric 255
}

# Define the Lookup Routes on all switches that require it
# First local
New-NetVirtualizationLookupRecord -VMName Red11 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.11 -MACAddress 00155D002009 -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $redGUID
New-NetVirtualizationLookupRecord -VMName Red22 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.22 -MACAddress 00155D002008 -ProviderAddress 172.16.0.2 -Rule TranslationMethodEncap -CustomerID $redGUID
New-NetVirtualizationLookupRecord -VMName Blue11 -VirtualSubnetID 7788990 -CustomerAddress 192.168.1.11 -MACAddress 00155D00200B -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $blueGUID
New-NetVirtualizationLookupRecord -VMName Blue22 -VirtualSubnetID 7788990 -CustomerAddress 192.168.1.22 -MACAddress 00155D00200A -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $blueGUID

# Then remote
Invoke-Command -Session $waldorf -ArgumentList $redGUID, $blueGUID {
Param($redGUID, $blueGUID)
New-NetVirtualizationLookupRecord -VMName Red11 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.11 -MACAddress 00155D002009 -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $redGUID
New-NetVirtualizationLookupRecord -VMName Red22 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.22 -MACAddress 00155D002008 -ProviderAddress 172.16.0.2 -Rule TranslationMethodEncap -CustomerID $redGUID
New-NetVirtualizationLookupRecord -VMName Blue11 -VirtualSubnetID 7788990 -CustomerAddress 192.168.1.11 -MACAddress 00155D00200B -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $blueGUID
New-NetVirtualizationLookupRecord -VMName Blue22 -VirtualSubnetID 7788990 -CustomerAddress 192.168.1.22 -MACAddress 00155D00200A -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $blueGUID
}

# Ping is working again

# Take a look and the LookupRoutes
Get-NetVirtualizationLookupRecord | ft CustomerAddress, CustomerID, Provideraddress, macaddress, rule, vmname -AutoSize

# Break one of the routes, this can be due to a VM HA event, or a mis-configuraiton.
Remove-NetVirtualizationLookupRecord -CustomerAddress 192.168.1.22 -VMName Red22
New-NetVirtualizationLookupRecord -VMName Red22 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.22 -MACAddress 00155D002008 -ProviderAddress 172.16.0.1 -Rule TranslationMethodEncap -CustomerID $redGUID

# Now the route is incorrect, what happens? (check ping)

# Repair the route

Remove-NetVirtualizationLookupRecord -CustomerAddress 192.168.1.22 -VMName Red22
New-NetVirtualizationLookupRecord -VMName Red22 -VirtualSubnetID 445566 -CustomerAddress 192.168.1.22 -MACAddress 00155D002008 -ProviderAddress 172.16.0.2 -Rule TranslationMethodEncap -CustomerID $redGUID

Server 2012 Windows Network Virtualization part 6 Lookup Routes

I am finally at the last part that makes this all work.

If you have defined everything so far you have VM traffic routing between VMs on the same Virtual Switch (and it is isolated like a tiny VLAN) but nothing more.  No traffic will travel from VM to VM across hosts until we define the Lookup Route that tells that packet where to go.

here is the command in a nutshell:

New-NetVirtualizationLookupRecord -VirtualSubnetID 5001 -CustomerAddress 192.168.104.10 -MACAddress 00155D002001 -ProviderAddress 10.235.0.31 -Rule TranslationMethodNat -CustomerID "{2ad17590-33b5-45fc-ad3a-90e5ff9b017f}" -VMName VMPXE

The VirtualSubnetID, the CA (the IP of the VM), the MAC of the VM, the PA that the CA is associated with, the type of network virtualization to use, the Routing Domain (Customer ID) and the name of the VM (really for management benefits – it does not have to match anything).

The –Rules are TranslationMethodNat (for IP Rewrite) or TranslationMethodEncap for (NVGRE encapsulation).  This is that decision I mentioned before.

If your traffic is all IP based and can handle segmentation then by all means use encapsulation (also, your physical devises must support it, most do, but as long as they don’t somehow do something with the virtual subnet ID you should be good).

Otherwise fall back to Nat (IP Rewrite).  But, remember that if you use this you need one PA for each CA, so you need to add PA addresses each time you add a VM.  And don’t worry, you can define multiples on the physical NIC of the virtual switch.

Here I have a script to set up encapsulation on a static network (where I don’t have to worry about VM change).

Here are my assumptions:

1. The PA is defined and I am using one for all VMs
2. The VMs are running and reporting an IP address to Hyper-V
3. I write an output file to run on the ‘other’ Hyper-V Servers to propagate the Lookup Routes.
4. If the IP is 0.0.0.0 I ignore it.

There is a lot going on here and the screen does not format it well, be warned.

$paAddress = Get-NetVirtualizationProviderAddress

$otherHostsFile = $env:Public + "\Documents\" + "RunOnHostsOtherThan_" + $env:COMPUTERNAME + ".ps1"
Out-File -Force $otherHostsFile  # this is the propagation file

$vms = Get-VM -Name *

foreach ($i in $vms) {
    $i.VMName

    if ($i.NetworkAdapters.IPAddresses -ne $null) {
        $vmIpv4 = $i.NetworkAdapters.IPAddresses | where { $_.contains(".") }
        $vmIpv6 = $i.NetworkAdapters.IPAddresses | where { $_.contains(":") }
    }
    $vmNic = Get-VMNetworkAdapter -VM $i

    "    " + $vmNic.Name + " -> " + $vmNic.SwitchName

    if ($vmIpv4 -ne $NULL) {
        New-NetVirtualizationLookupRecord -VirtualSubnetID $vSubnetId -CustomerAddress $vmIpv4 -MACAddress $vmNic.MACAddress -ProviderAddress $paAddress.ProviderAddress -Rule TranslationMethodEncap -CustomerID $routingDomainId -VMName $vmNic.VMName
        Set-VMNetworkAdapter -VirtualSubnetId $vSubnetId -VMNetworkAdapter $vmNic
        ("New-NetVirtualizationLookupRecord -VirtualSubnetID " + $vSubnetId + " -CustomerAddress " + $vmIpv4 + " -MACAddress " + $vmNic.MACAddress + " -ProviderAddress " + $paAddress.ProviderAddress+ " -Rule TranslationMethodEncap -CustomerID `"" + $routingDomainId + "`" -VMName " + $vmNic.VMName) | Out-File -NoClobber -Append -FilePath $otherHostsFile

    }
    <#    If ($vmIpv6 -ne $NULL) {
        New-NetVirtualizationLookupRecord -VirtualSubnetID $vSubnetId -CustomerAddress $vmIpv6 -MACAddress $vmNic.MACAddress -ProviderAddress $paAddress.ProviderAddress -Rule TranslationMethodEncap -CustomerID $routingDomainId -VMName $vmNic.VMName
        Set-VMNetworkAdapter -VirtualSubnetId $vSubnetId -VMNetworkAdapter $vmNic
        ("New-NetVirtualizationLookupRecord -VirtualSubnetID " + $vSubnetId + " -CustomerAddress " + $vmIpv6+ " -MACAddress " + $vmNic.MACAddress + " -ProviderAddress " + $paAddress.ProviderAddress+ " -Rule TranslationMethodEncap -CustomerID `"" + $routingDomainId + "`" -VMName " + $vmNic.VMName) | Out-File -NoClobber -Append -FilePath $otherHostsFile
    }#>

    Clear-Variable -name vmIpv4 -ErrorAction SilentlyContinue
    Clear-Variable -Name vmIpv6 -ErrorAction SilentlyContinue
}

 

I have another script to set up NAT and that takes a bit more due to the PA management.  A post of its own.

Server 2012 Windows Network Virtualization part 5 the Customer Address

I am not going to spend much if any time on this.

Do you recall this graphic:

 

All that you need to know is that the Customer Address must be a known thing.  You need to know the IP address of your VM, the MAC or your VM, and how you are going to forward your VM traffic (encapsulation or IP Rewrite) prior to the next step.

Remember, if you use IP Rewrite – each CA needs a unique PA.  If you use encapsulation you can have one PA support multiple CAs.

IP Rewrite gives high compatibility but is more to manage, Encapsulation is easier to manage but might not work in all cases (actually, I know cases where it fails).

The CA is the IP of the VM.  There can only be one LookupRoute defined per CA + VirtualSubnetID.  Remember, VirtualSubnetID is your isolation boundary, within the CustomerRoute, but cannot be used more than once within a datacenter.

This can also be thought of as the IP that defines the target route – as a remote CA is actually the target of any packet.

And, most important – in the WNV model, this is the IP that the customer brings with them.  There can be duplicates, but not for a single customer – as that would duplicate too many settings and create conflicting routing rules.

Server 2012 Windows Network Virtualization part 4 the PA address

In the previous post I set the basics.  I enabled WNV and left you hanging.

Lets begin with getting the concepts straight about the PA (ProviderAddress) and the CA (CustomerAddress).  If you think about the paradigm, these names reflect the owner of the respective subnet / address space.

The PA is an address that is owned by the Provider.  Also referred to as the Hoster or the Enterprise.  These addresses allow packets to be routed between Hyper-V Virtual Switches.

The CA is an address that is brought by the Customer.  Also referred to as a Tenant, or a cloud user.  These addresses are the addresses that the VMs have.  And need to be known when establishing the routing rules.

Lets just focus on the PA for the remainder of this post.

An important thing to understand (that I really did not get at first) is that the ProviderAddress (the PA) is unique and different from the Management IP of the Hyper-V Server.

I draw it this way:

Also, the ProviderAddress just needs to be routable to any other hosts where VMs reside that participate in a particular virtual network.

For example:

Your management subnet might be 10.10.3.x.  If you don’t need any more than 254 PA addresses then you can use 192.168.3.x (for example).

In this model 192.168 is not routable, so all the Hyper-V Extensible Switches need to be local to each other. 

I am going to talk Switches here, since that is what this is all about, not the Hyper-V Server – that is a different IP and different issue.

ProviderAddresses are all about switch to switch communication using a routable IP address.  So all the networking or routing rules that you already know apply.  Your choice of PA addresses combined with routing rules can restrict your virtual network to a datacenter, or allow it to span datacenters, or even to remote services.  It is all in how you define and build it.

Lets add that you can define multiple PAs.  And the PAs can be private, public, different subnets, etc.  The IP of the PA simply interacts with the routing rules at the physical / fabric layer of your network.

Now, lets mention the two primary types of virtual networks.  MSFT has used the terms NVGRE (or Encapsulated) and IPRewrite (or NAT) in various places.  The reason that I mention this is that it affects your PA options.

IPRewrite sets up a NAT rule.  This requires one PA per CA.  So, each VM has an IP address (the CA) and a corresponding PA.  The rules for IPRewrite is essentially a one to one NAT (no port redirection here like in a router this is IP translation).

NVGRE uses encapsulation.  This is the most flexible in how you define it.  As you can use one PA for multiple CAs.  And again, you can have multiple PAs.  Here are some possibilities.

The simplest model of one PA per virtual switch:

This is compatible with:  NVRGE (all virtual networks using NVGRE)

 

Another possibility is one PA per virtual network.  This would give a bit more flexibility in managing different tenants sharing Hyper-V Servers. Again, this would only support using NVGRE (encapsulation).

 

If you want to use IPRewrite you need one PA for each VM.

After all this, you want to define a ProviderAddress?

It is really simple.  In this example I am assuming one thing, you have only one External Virtual Switch.

$wnvIf = Get-NetAdapterBinding -ComponentID "ms_netwnv" | Where-Object { $_.Enabled -eq $true } | Get-netadapter

New-NetVirtualizationProviderAddress -InterfaceIndex $wnvIf.InterfaceIndex -ProviderAddress 192.168.100.3 -PrefixLength 24

Next, we will tackle the CA space.

Server 2012 Windows Network Virtualization part 3 the Virtual Subnet Id

First of all, I want to say that I am only describing these concepts in a way that I think I can build from one concept to the next.

The order that you define these various things in does not matter.  However, the data does not flow until it is all in place.

In an attempt to better relate the concept of where these various settings reside I have drawn the following diagram:

 

In this article lets talk about the Virtual Subnet Id in a bit of detail. This is a relatively simple setting.

This is a setting of the port of the Virtual Switch.  However, it is presented to you as a property of the Virtual Network Interface of a VM. 

It is defined in this way as it makes sense to want these settings to remain with the VM, not the virtual switch port.  So it just helps the brain to keep things straight when the UI calls this a vNIC or virtual NIC.

Now, down to business.

If you list the properties of a VM NetworkAdapter you will see the VirtualSubnetId. 

PS C:\Users\administrator.BRIANEH> Get-VMNetworkAdapter -VMName VM2 | fl

Name                     : VM2
Id                       : Microsoft:D9B02F4B-47B1-4A88-A20B-E763CC97209A\71803
                           2DA-6EFB-4EF4-8152-9A2FBDA945BF\0
IsLegacy                 : True
IsManagementOs           : False
ComputerName             : BJELKS5
VMName                   : VM2
VMId                     : d9b02f4b-47b1-4a88-a20b-e763cc97209a
SwitchName               : Virtual Network0
SwitchId                 : a6891eb9-bb01-45e5-bf20-e8a450fb0135
Connected                : True
PoolName                 :
MacAddress               : 001DD8B71C01
DynamicMacAddressEnabled : False
MacAddressSpoofing       : Off
AllowTeaming             : Off
RouterGuard              : Off
DhcpGuard                : Off
PortMirroringMode        : None
IeeePriorityTag          : Off
VirtualSubnetId          : 11657562
VMQWeight                : 0
VMQUsage                 : 0
IOVWeight                : 0
IOVUsage                 : 0
IovQueuePairsRequested   : 1
IovQueuePairsAssigned    : 0
IOVInterruptModeration   : Default
IPsecOffloadMaxSA        : 0
IPsecOffloadSAUsage      : 0
VFDataPathActive         : False
MaximumBandwidth         :
MinimumBandwidthAbsolute :
MinimumBandwidthWeight   :
BandwidthPercentage      : 0%
MandatoryFeatureId       : {}
MandatoryFeatureName     : {}
Status                   : {Ok}
IPAddresses              : {192.168.1.3, fe80::78aa:947f:8718:d231}

However, if you look in the Hyper-V Manager GUI you will never see this property.  This is only available only through PowerShell or WMI as it is considered a far advanced setting.

If you did no more than set the VirtualSubnetId of of a bunch of VMs to the same VirtualSubnetId you would have the equivalent of a VLAN tag that is local to the Virtual Switch.  The virtual switch would route the VirtualSubnetIds internally and sort them out, very much like a VLAN tag.

This concept alone can be handy.

But, for the traffic to leave the local virtual switch you need the other settings in place and defined for the Network Virtualization module.

So, how do you set this?  Well, the Set-VMNetworkAdapter cmdlet supports multiple options.

If you have the VMNetworkAdapter object you can do this: 

Set-VMNetworkAdapter -VirtualSubnetId $vSubnetId -VMNetworkAdapter $vmNic

If you want to set all the VMNetworkAdapters on your Hyper-V Server to the same VirtualSubnetId you can do this: 

Get-VMNetworkAdapter * | Set-VMNetworkAdapter –VirtualSubnetId $vSubnetId

The Key is that the VirtualSubnetId is higher than the VLAN tag top limit of 4095, it goes from 4096 to 16,777,215. A VMNetworkAdapter can only have one VirtualSubnetId at any time.  Setting this to 0 is the same as not having it set (just like VLAN tags).

One more thing is that the VirtualSubnetID must only be used once in the datacenter.  In other words Customer A and B in the Big Hosted Datacenter could not both use the VirtualSubnetID of 5001 even though they have unique Routing Domains.

The very last item to know about the VirtualSubnetID is that in the NVGRE internet draft it is mentioned that this can be utilized by the physical switching devices to more efficiently route traffic.  This opens a huge potential.  And if the fabric can self learn routes (the fabric does the forwarding, not the virtual switch), maybe some of the upcoming details to maintain routes could be avoided.  Huge potential.

Server 2012 Windows Network Virtualization part 2 the Customer Route

So, at this point Network Virtualization is turned on.  Yea!  Now we can use it.  But we have to define and know a few things.

One thing we need is a Customer Route also referred to as a Routing Domain (an easier term to use I think, but I will stay consistent with the API terms).

A Customer Route can be described in a couple different ways.  And how I describe this affects how I describe other features later on.  But the easiest way to describe a Routing Domain / Customer Route is that it is a single Virtual Network.

A Customer Route is a routing container, or a routing domain.  It can contain multiple virtual subnets.  In the Windows Network Virtualization implementation a Customer Route should represent a single Customer or Tenant.  And this customer can then define multiple VLANs within this route.

For example:

Say that you want to translate your current network into a Virtual Network.  You have a flat topology using various subnets divided by VLANs.

IP addresses with no VLAN tags are client desktops and services they consume.  VLAN 10 is Server management interfaces.  VLAN 20 is application server to database server traffic.  And so on.

If I was a Hosting provider using Server 2012.  I would create one CustomerRoute for you.  Within this Route I would establish at least two virtual subnets.  One for application server to database traffic, one to isolate the management interfaces.  Here is the hard part, how to handle the client to application server traffic.  If the application server is exposed on a public IP then I wouldn’t add a virtual subnet.  If the application servers are in your datacenter and I am hosting the database servers then I establish Network Virtualization rules so that your ProviderAddress(es) can reach mine (more on this later).

I could go on an on and as I move through the articles I hope the picture of possibilities and dependencies becomes clear.

Regardless what I do within a CustomerRoute, it is yours.  It is a single routing container that is isolated from all others.  Within this there needs to be at least one VirtualSubnetId and there can be many (more on that next).

Lets define one.

The Customer Route is defined using a GUID.  This GUID is referred to as the RoutingDomainId.  You have to derive one.  Oh, you also need to know what the CA addresses of the VMs will be (the IP subnet that will be used within a particular virtual subnet).

So, lets establish our constants:

$customerName = "Atomic Widgets"
$routingDomainId = "{2ad17590-33b5-45fc-ad3a-90e5ff9b017f}" #Note the brackets in the string, that is important.
$vSubnetId = "5001"
$caDestinationPrefix = "192.168.104.0/24”

$caNexthopGw = "0.0.0.0"

What I have done here is that I state that VirtualSubnetId 5001 contains VMs with IP addresses within the 192.168.104.x class C subnet.

I then need to define this CustomerRoute on each Hyper-V Server where there will be VMs participating in this CustomerRoute / Virtual Network.

New-NetVirtualizationCustomerRoute -RoutingDomainID $routingDomainId -VirtualSubnetID $vSubnetId -DestinationPrefix $caDestinationPrefix -NextHop $caNexthopGw -Metric 255

If I did no more than add this Customer Route, and then created VMs on this Hyper-V Server with the VirtualSubnetId of 5001 I will have the equivalent of my own (local) isolated VLAN.  The traffic is 100% local to my Hyper-V Server even though this is an External Virtual Switch.  It takes the establishment of some additional settings and routing rules for the VM traffic to exit the virtual switch and know where to go.

Actually, I don’t even need to set the Customer Route to have my own private VLAN 5001 within one Virtual Switch.  I really only need to set the virtual subnet id on the virtual NIC of the VMs.

Okay, enough of that, lets go back to my Virtual Network that is my CustomerRoute / RoutingDomain…

What if my Virtual Network has multiple VirtualSubnetIds (multiple little internal VLANs)?

Well, you simply add additional CustomerRoute definitions.  One for each virtual subnet id.

What does this look like if there are multiple virtual subnets?

PS C:\Users\administrator.BRIANEH> Get-NetVirtualizationCustomerRoute

RoutingDomainID   : {99B3D49E-9281-4C8E-81A0-8DC2257FB7EE}
VirtualSubnetID   : 7636197
DestinationPrefix : 192.168.4.0/24
NextHop           : 0.0.0.0
Metric            : 9

RoutingDomainID   : {99B3D49E-9281-4C8E-81A0-8DC2257FB7EE}
VirtualSubnetID   : 9889980
DestinationPrefix : 192.168.3.0/24
NextHop           : 0.0.0.0
Metric            : 9

RoutingDomainID   : {99B3D49E-9281-4C8E-81A0-8DC2257FB7EE}
VirtualSubnetID   : 4139103
DestinationPrefix : 192.168.2.0/24
NextHop           : 0.0.0.0
Metric            : 9

RoutingDomainID   : {99B3D49E-9281-4C8E-81A0-8DC2257FB7EE}
VirtualSubnetID   : 11657562
DestinationPrefix : 192.168.1.0/24
NextHop           : 0.0.0.0
Metric            : 9

The same RoutingDomainID is used, but each subnet is defined with its destination prefix(s).  There will be more on that later.

*Just a note, I am working on a good graphical way to describe this all.

Server 2012 Windows Network Virtualization part 1 of many

I have spent quite an extensive amount of time working with the Network Virtualization that is in Server 2012.
Most folks that are interested in this have caught the article on Jeff Snover's blog. And have possibly seen a few TechEd presentations that keep working the same graphics.  And, if you explore a bit you can find some interesting articles from folks in the networking community like over at NetworkHeresy.com about how NVGRE is a ‘right thing’.  And a primer on NVGRE over at SearchNetworking.com.
Oh, and lets not forget the Network Virtualization Survival Guide by the MSFT folks.
That all sets the basis for the technology.  Now the interesting bits.  The implementation.
First of all, lets get a big question out of the way:  Is SCVMM required to manage Windows Network Virtualization?  No, it isn’t.  However, you want some management layer to manage the complexities of keeping all of the routes and what not up to date.   If you look at Jeff Snover’s graphics, there is this whole set of PA and CA addresses that need to be kept straight.  And then there is the LookupRecords.
Really, this is not too bad in a static environment.  If the VMS never move or change, you set all these rules once and never touch them again.  However, if you move a VM (and Server 2012 makes that super easy) then you need to update the proper records or that VM is off the network.  SCVMM 2012 has a similar challenge of keeping LookupRecords and PAs updated if a VM is moved by Failover Clustering due to an HA event that it did not do.
In other words, this is not for the faint of heart.  There is actually a lot to this.  If you move beyond an environment where your VMs are static then you need to maintain these rules.  This is what management layers (like SCVMM, CloudStack) do for you, manage the complex stuff.
Okay, thanks for the warning, now what?
Step one:
Enable Windows Network Virtualization (WNV).  This is a binding that is applied to the NIC that you External Virtual Switch is bound to.  This can be a physical NIC, it can be an LBFO NIC team.  Either way, it is the network adapter that your External Virtual Switch uses to exit the server.
This also means that if you have multiple virtual networks or multiple interfaces that you can pick and choose and it is not some global setting.
If you have one External Virtual Switch this is fairly easy:

$vSwitch = Get-VMSwitch -SwitchType External
# Check if Network Virtualization is bound
# This could be done by checking for the binding and seeing if it is enabled
ForEach-Object -InputObject $vSwitch {
if ((Get-NetAdapterBinding -ComponentID "ms_netwnv" -InterfaceDescription $_.NetAdapterInterfaceDescription).Enabled -eq $false){
  # Lets enable it
  Enable-NetAdapterBinding -InterfaceDescription $_.NetAdapterInterfaceDescription -ComponentID "ms_netwnv"
}
}

I have a ForEach in the example above.  In case $vSwitch is an array, I assume I want to enable it on all virtual switches.  We will deal with the other details later.
Before me move on, I need to know some constants.  Such as my Tenant (Customer).  And some topology.  We will get there as we work through defining everything.
More to come.  Lots to learn about PA, CA, lookup routes, virtual subnet ids, and customer routes.  Lots of concepts, options, and rules to keep straight.