Server 2012 Windows Network Virtualization part 6 Lookup Routes

I am finally at the last part that makes this all work.

If you have defined everything so far you have VM traffic routing between VMs on the same Virtual Switch (and it is isolated like a tiny VLAN) but nothing more.  No traffic will travel from VM to VM across hosts until we define the Lookup Route that tells that packet where to go.

here is the command in a nutshell:

New-NetVirtualizationLookupRecord -VirtualSubnetID 5001 -CustomerAddress 192.168.104.10 -MACAddress 00155D002001 -ProviderAddress 10.235.0.31 -Rule TranslationMethodNat -CustomerID "{2ad17590-33b5-45fc-ad3a-90e5ff9b017f}" -VMName VMPXE

The VirtualSubnetID, the CA (the IP of the VM), the MAC of the VM, the PA that the CA is associated with, the type of network virtualization to use, the Routing Domain (Customer ID) and the name of the VM (really for management benefits – it does not have to match anything).

The –Rules are TranslationMethodNat (for IP Rewrite) or TranslationMethodEncap for (NVGRE encapsulation).  This is that decision I mentioned before.

If your traffic is all IP based and can handle segmentation then by all means use encapsulation (also, your physical devises must support it, most do, but as long as they don’t somehow do something with the virtual subnet ID you should be good).

Otherwise fall back to Nat (IP Rewrite).  But, remember that if you use this you need one PA for each CA, so you need to add PA addresses each time you add a VM.  And don’t worry, you can define multiples on the physical NIC of the virtual switch.

Here I have a script to set up encapsulation on a static network (where I don’t have to worry about VM change).

Here are my assumptions:

1. The PA is defined and I am using one for all VMs
2. The VMs are running and reporting an IP address to Hyper-V
3. I write an output file to run on the ‘other’ Hyper-V Servers to propagate the Lookup Routes.
4. If the IP is 0.0.0.0 I ignore it.

There is a lot going on here and the screen does not format it well, be warned.

$paAddress = Get-NetVirtualizationProviderAddress

$otherHostsFile = $env:Public + "\Documents\" + "RunOnHostsOtherThan_" + $env:COMPUTERNAME + ".ps1"
Out-File -Force $otherHostsFile  # this is the propagation file

$vms = Get-VM -Name *

foreach ($i in $vms) {
    $i.VMName

    if ($i.NetworkAdapters.IPAddresses -ne $null) {
        $vmIpv4 = $i.NetworkAdapters.IPAddresses | where { $_.contains(".") }
        $vmIpv6 = $i.NetworkAdapters.IPAddresses | where { $_.contains(":") }
    }
    $vmNic = Get-VMNetworkAdapter -VM $i

    "    " + $vmNic.Name + " -> " + $vmNic.SwitchName

    if ($vmIpv4 -ne $NULL) {
        New-NetVirtualizationLookupRecord -VirtualSubnetID $vSubnetId -CustomerAddress $vmIpv4 -MACAddress $vmNic.MACAddress -ProviderAddress $paAddress.ProviderAddress -Rule TranslationMethodEncap -CustomerID $routingDomainId -VMName $vmNic.VMName
        Set-VMNetworkAdapter -VirtualSubnetId $vSubnetId -VMNetworkAdapter $vmNic
        ("New-NetVirtualizationLookupRecord -VirtualSubnetID " + $vSubnetId + " -CustomerAddress " + $vmIpv4 + " -MACAddress " + $vmNic.MACAddress + " -ProviderAddress " + $paAddress.ProviderAddress+ " -Rule TranslationMethodEncap -CustomerID `"" + $routingDomainId + "`" -VMName " + $vmNic.VMName) | Out-File -NoClobber -Append -FilePath $otherHostsFile

    }
    <#    If ($vmIpv6 -ne $NULL) {
        New-NetVirtualizationLookupRecord -VirtualSubnetID $vSubnetId -CustomerAddress $vmIpv6 -MACAddress $vmNic.MACAddress -ProviderAddress $paAddress.ProviderAddress -Rule TranslationMethodEncap -CustomerID $routingDomainId -VMName $vmNic.VMName
        Set-VMNetworkAdapter -VirtualSubnetId $vSubnetId -VMNetworkAdapter $vmNic
        ("New-NetVirtualizationLookupRecord -VirtualSubnetID " + $vSubnetId + " -CustomerAddress " + $vmIpv6+ " -MACAddress " + $vmNic.MACAddress + " -ProviderAddress " + $paAddress.ProviderAddress+ " -Rule TranslationMethodEncap -CustomerID `"" + $routingDomainId + "`" -VMName " + $vmNic.VMName) | Out-File -NoClobber -Append -FilePath $otherHostsFile
    }#>

    Clear-Variable -name vmIpv4 -ErrorAction SilentlyContinue
    Clear-Variable -Name vmIpv6 -ErrorAction SilentlyContinue
}

 

I have another script to set up NAT and that takes a bit more due to the PA management.  A post of its own.

Server 2012 Windows Network Virtualization part 5 the Customer Address

I am not going to spend much if any time on this.

Do you recall this graphic:

 

All that you need to know is that the Customer Address must be a known thing.  You need to know the IP address of your VM, the MAC or your VM, and how you are going to forward your VM traffic (encapsulation or IP Rewrite) prior to the next step.

Remember, if you use IP Rewrite – each CA needs a unique PA.  If you use encapsulation you can have one PA support multiple CAs.

IP Rewrite gives high compatibility but is more to manage, Encapsulation is easier to manage but might not work in all cases (actually, I know cases where it fails).

The CA is the IP of the VM.  There can only be one LookupRoute defined per CA + VirtualSubnetID.  Remember, VirtualSubnetID is your isolation boundary, within the CustomerRoute, but cannot be used more than once within a datacenter.

This can also be thought of as the IP that defines the target route – as a remote CA is actually the target of any packet.

And, most important – in the WNV model, this is the IP that the customer brings with them.  There can be duplicates, but not for a single customer – as that would duplicate too many settings and create conflicting routing rules.

Server 2012 Windows Network Virtualization part 4 the PA address

In the previous post I set the basics.  I enabled WNV and left you hanging.

Lets begin with getting the concepts straight about the PA (ProviderAddress) and the CA (CustomerAddress).  If you think about the paradigm, these names reflect the owner of the respective subnet / address space.

The PA is an address that is owned by the Provider.  Also referred to as the Hoster or the Enterprise.  These addresses allow packets to be routed between Hyper-V Virtual Switches.

The CA is an address that is brought by the Customer.  Also referred to as a Tenant, or a cloud user.  These addresses are the addresses that the VMs have.  And need to be known when establishing the routing rules.

Lets just focus on the PA for the remainder of this post.

An important thing to understand (that I really did not get at first) is that the ProviderAddress (the PA) is unique and different from the Management IP of the Hyper-V Server.

I draw it this way:

Also, the ProviderAddress just needs to be routable to any other hosts where VMs reside that participate in a particular virtual network.

For example:

Your management subnet might be 10.10.3.x.  If you don’t need any more than 254 PA addresses then you can use 192.168.3.x (for example).

In this model 192.168 is not routable, so all the Hyper-V Extensible Switches need to be local to each other. 

I am going to talk Switches here, since that is what this is all about, not the Hyper-V Server – that is a different IP and different issue.

ProviderAddresses are all about switch to switch communication using a routable IP address.  So all the networking or routing rules that you already know apply.  Your choice of PA addresses combined with routing rules can restrict your virtual network to a datacenter, or allow it to span datacenters, or even to remote services.  It is all in how you define and build it.

Lets add that you can define multiple PAs.  And the PAs can be private, public, different subnets, etc.  The IP of the PA simply interacts with the routing rules at the physical / fabric layer of your network.

Now, lets mention the two primary types of virtual networks.  MSFT has used the terms NVGRE (or Encapsulated) and IPRewrite (or NAT) in various places.  The reason that I mention this is that it affects your PA options.

IPRewrite sets up a NAT rule.  This requires one PA per CA.  So, each VM has an IP address (the CA) and a corresponding PA.  The rules for IPRewrite is essentially a one to one NAT (no port redirection here like in a router this is IP translation).

NVGRE uses encapsulation.  This is the most flexible in how you define it.  As you can use one PA for multiple CAs.  And again, you can have multiple PAs.  Here are some possibilities.

The simplest model of one PA per virtual switch:

This is compatible with:  NVRGE (all virtual networks using NVGRE)

 

Another possibility is one PA per virtual network.  This would give a bit more flexibility in managing different tenants sharing Hyper-V Servers. Again, this would only support using NVGRE (encapsulation).

 

If you want to use IPRewrite you need one PA for each VM.

After all this, you want to define a ProviderAddress?

It is really simple.  In this example I am assuming one thing, you have only one External Virtual Switch.

$wnvIf = Get-NetAdapterBinding -ComponentID "ms_netwnv" | Where-Object { $_.Enabled -eq $true } | Get-netadapter

New-NetVirtualizationProviderAddress -InterfaceIndex $wnvIf.InterfaceIndex -ProviderAddress 192.168.100.3 -PrefixLength 24

Next, we will tackle the CA space.

Server 2012 Windows Network Virtualization part 3 the Virtual Subnet Id

First of all, I want to say that I am only describing these concepts in a way that I think I can build from one concept to the next.

The order that you define these various things in does not matter.  However, the data does not flow until it is all in place.

In an attempt to better relate the concept of where these various settings reside I have drawn the following diagram:

 

In this article lets talk about the Virtual Subnet Id in a bit of detail. This is a relatively simple setting.

This is a setting of the port of the Virtual Switch.  However, it is presented to you as a property of the Virtual Network Interface of a VM. 

It is defined in this way as it makes sense to want these settings to remain with the VM, not the virtual switch port.  So it just helps the brain to keep things straight when the UI calls this a vNIC or virtual NIC.

Now, down to business.

If you list the properties of a VM NetworkAdapter you will see the VirtualSubnetId. 

PS C:\Users\administrator.BRIANEH> Get-VMNetworkAdapter -VMName VM2 | fl

Name                     : VM2
Id                       : Microsoft:D9B02F4B-47B1-4A88-A20B-E763CC97209A\71803
                           2DA-6EFB-4EF4-8152-9A2FBDA945BF\0
IsLegacy                 : True
IsManagementOs           : False
ComputerName             : BJELKS5
VMName                   : VM2
VMId                     : d9b02f4b-47b1-4a88-a20b-e763cc97209a
SwitchName               : Virtual Network0
SwitchId                 : a6891eb9-bb01-45e5-bf20-e8a450fb0135
Connected                : True
PoolName                 :
MacAddress               : 001DD8B71C01
DynamicMacAddressEnabled : False
MacAddressSpoofing       : Off
AllowTeaming             : Off
RouterGuard              : Off
DhcpGuard                : Off
PortMirroringMode        : None
IeeePriorityTag          : Off
VirtualSubnetId          : 11657562
VMQWeight                : 0
VMQUsage                 : 0
IOVWeight                : 0
IOVUsage                 : 0
IovQueuePairsRequested   : 1
IovQueuePairsAssigned    : 0
IOVInterruptModeration   : Default
IPsecOffloadMaxSA        : 0
IPsecOffloadSAUsage      : 0
VFDataPathActive         : False
MaximumBandwidth         :
MinimumBandwidthAbsolute :
MinimumBandwidthWeight   :
BandwidthPercentage      : 0%
MandatoryFeatureId       : {}
MandatoryFeatureName     : {}
Status                   : {Ok}
IPAddresses              : {192.168.1.3, fe80::78aa:947f:8718:d231}

However, if you look in the Hyper-V Manager GUI you will never see this property.  This is only available only through PowerShell or WMI as it is considered a far advanced setting.

If you did no more than set the VirtualSubnetId of of a bunch of VMs to the same VirtualSubnetId you would have the equivalent of a VLAN tag that is local to the Virtual Switch.  The virtual switch would route the VirtualSubnetIds internally and sort them out, very much like a VLAN tag.

This concept alone can be handy.

But, for the traffic to leave the local virtual switch you need the other settings in place and defined for the Network Virtualization module.

So, how do you set this?  Well, the Set-VMNetworkAdapter cmdlet supports multiple options.

If you have the VMNetworkAdapter object you can do this: 

Set-VMNetworkAdapter -VirtualSubnetId $vSubnetId -VMNetworkAdapter $vmNic

If you want to set all the VMNetworkAdapters on your Hyper-V Server to the same VirtualSubnetId you can do this: 

Get-VMNetworkAdapter * | Set-VMNetworkAdapter –VirtualSubnetId $vSubnetId

The Key is that the VirtualSubnetId is higher than the VLAN tag top limit of 4095, it goes from 4096 to 16,777,215. A VMNetworkAdapter can only have one VirtualSubnetId at any time.  Setting this to 0 is the same as not having it set (just like VLAN tags).

One more thing is that the VirtualSubnetID must only be used once in the datacenter.  In other words Customer A and B in the Big Hosted Datacenter could not both use the VirtualSubnetID of 5001 even though they have unique Routing Domains.

The very last item to know about the VirtualSubnetID is that in the NVGRE internet draft it is mentioned that this can be utilized by the physical switching devices to more efficiently route traffic.  This opens a huge potential.  And if the fabric can self learn routes (the fabric does the forwarding, not the virtual switch), maybe some of the upcoming details to maintain routes could be avoided.  Huge potential.

Server 2012 Windows Network Virtualization part 2 the Customer Route

So, at this point Network Virtualization is turned on.  Yea!  Now we can use it.  But we have to define and know a few things.

One thing we need is a Customer Route also referred to as a Routing Domain (an easier term to use I think, but I will stay consistent with the API terms).

A Customer Route can be described in a couple different ways.  And how I describe this affects how I describe other features later on.  But the easiest way to describe a Routing Domain / Customer Route is that it is a single Virtual Network.

A Customer Route is a routing container, or a routing domain.  It can contain multiple virtual subnets.  In the Windows Network Virtualization implementation a Customer Route should represent a single Customer or Tenant.  And this customer can then define multiple VLANs within this route.

For example:

Say that you want to translate your current network into a Virtual Network.  You have a flat topology using various subnets divided by VLANs.

IP addresses with no VLAN tags are client desktops and services they consume.  VLAN 10 is Server management interfaces.  VLAN 20 is application server to database server traffic.  And so on.

If I was a Hosting provider using Server 2012.  I would create one CustomerRoute for you.  Within this Route I would establish at least two virtual subnets.  One for application server to database traffic, one to isolate the management interfaces.  Here is the hard part, how to handle the client to application server traffic.  If the application server is exposed on a public IP then I wouldn’t add a virtual subnet.  If the application servers are in your datacenter and I am hosting the database servers then I establish Network Virtualization rules so that your ProviderAddress(es) can reach mine (more on this later).

I could go on an on and as I move through the articles I hope the picture of possibilities and dependencies becomes clear.

Regardless what I do within a CustomerRoute, it is yours.  It is a single routing container that is isolated from all others.  Within this there needs to be at least one VirtualSubnetId and there can be many (more on that next).

Lets define one.

The Customer Route is defined using a GUID.  This GUID is referred to as the RoutingDomainId.  You have to derive one.  Oh, you also need to know what the CA addresses of the VMs will be (the IP subnet that will be used within a particular virtual subnet).

So, lets establish our constants:

$customerName = "Atomic Widgets"
$routingDomainId = "{2ad17590-33b5-45fc-ad3a-90e5ff9b017f}" #Note the brackets in the string, that is important.
$vSubnetId = "5001"
$caDestinationPrefix = "192.168.104.0/24”

$caNexthopGw = "0.0.0.0"

What I have done here is that I state that VirtualSubnetId 5001 contains VMs with IP addresses within the 192.168.104.x class C subnet.

I then need to define this CustomerRoute on each Hyper-V Server where there will be VMs participating in this CustomerRoute / Virtual Network.

New-NetVirtualizationCustomerRoute -RoutingDomainID $routingDomainId -VirtualSubnetID $vSubnetId -DestinationPrefix $caDestinationPrefix -NextHop $caNexthopGw -Metric 255

If I did no more than add this Customer Route, and then created VMs on this Hyper-V Server with the VirtualSubnetId of 5001 I will have the equivalent of my own (local) isolated VLAN.  The traffic is 100% local to my Hyper-V Server even though this is an External Virtual Switch.  It takes the establishment of some additional settings and routing rules for the VM traffic to exit the virtual switch and know where to go.

Actually, I don’t even need to set the Customer Route to have my own private VLAN 5001 within one Virtual Switch.  I really only need to set the virtual subnet id on the virtual NIC of the VMs.

Okay, enough of that, lets go back to my Virtual Network that is my CustomerRoute / RoutingDomain…

What if my Virtual Network has multiple VirtualSubnetIds (multiple little internal VLANs)?

Well, you simply add additional CustomerRoute definitions.  One for each virtual subnet id.

What does this look like if there are multiple virtual subnets?

PS C:\Users\administrator.BRIANEH> Get-NetVirtualizationCustomerRoute

RoutingDomainID   : {99B3D49E-9281-4C8E-81A0-8DC2257FB7EE}
VirtualSubnetID   : 7636197
DestinationPrefix : 192.168.4.0/24
NextHop           : 0.0.0.0
Metric            : 9

RoutingDomainID   : {99B3D49E-9281-4C8E-81A0-8DC2257FB7EE}
VirtualSubnetID   : 9889980
DestinationPrefix : 192.168.3.0/24
NextHop           : 0.0.0.0
Metric            : 9

RoutingDomainID   : {99B3D49E-9281-4C8E-81A0-8DC2257FB7EE}
VirtualSubnetID   : 4139103
DestinationPrefix : 192.168.2.0/24
NextHop           : 0.0.0.0
Metric            : 9

RoutingDomainID   : {99B3D49E-9281-4C8E-81A0-8DC2257FB7EE}
VirtualSubnetID   : 11657562
DestinationPrefix : 192.168.1.0/24
NextHop           : 0.0.0.0
Metric            : 9

The same RoutingDomainID is used, but each subnet is defined with its destination prefix(s).  There will be more on that later.

*Just a note, I am working on a good graphical way to describe this all.