Showing posts with label networking. Show all posts
Showing posts with label networking. Show all posts

Sunday, May 15, 2016

The Division and delta 20010186

There is no doubt that if you search on The Division for the Mike or Delta error messages you will uncover a sordid history of unhappy users that began with the beta and continues.

My son recently invested in this game only to pop it into the Xbox to be greeted by the "delta 20010186" error.

tl:dr
  1. enable uPNP
  2. If you have a router behind your modem, put your modem in bridge mode
  3. Rerun the Xbox multiplayer network test (see the link below) - UBISoft requires this to test as 'open'
The long winded explanation:

If you search on this you will find all kinds of guessed at solutions.  From punching port forwarding rules in your firewall to putting your Xbox wide open for attack in a DMZ (seriously, these are the suggestions from UBISoft for Xbox connectivity)

Considering the number of folks that run into this issue, I really don't think UBIsoft understands it.

After thinking about this and spending hours troubleshooting, my conclusion is this: UBISoft blocks the game working if there is any reduction in the multi-player experience at all.

If you read the Xbox network descriptions, there should be grades in the experience where only certain features of the game don't function if the connectivity status is reduced from the full set.  And I can only speculate that UBISoft chose to not follow this guideline (could be by choose or ignorance - the result is the same).

There is a mysterious black box type issue on the Xbox side that puts the pieces together.
It is rooted in the output of the Xbox multiplayer network test and the resulting "NAT type".
This is not a setting that any human can impact, nor any resetting, cryptic combinations of movement through the settings menus or anything else can reliably impact.

From what I can tell, there are a couple issues going on here.

1. There is an expectation that uPNP is enabled on your router.

The multiplayer mode requires communication back to your Xbox. 
This is why there are port forwarding rules being suggested, and DMZ placement.  If uPNP is not available or not working, this this would solve that issue.

Adding to this is claims that I uncovered that stated that Xbox historically had issues with properly making the uPNP router when it is set to 'InstantOn' power mode.
Again, uPNP not working for a different reason.

Wrap all of these together and you have a combination of things that all impact connectivity back and the Xbox test for multiplayer connectivity puts you in the Moderate or Severe NAT bucket.

2. Beware the double NAT

No matter the setting on your router, if you have put yourself into a double NAT configuration, you will never be tested as 'open' by Xbox.

What is this double NAT?  It is when you hop through two IP address changes on your end of the network.
Frequently a double NAT happens in the IT world to isolate front end web servers or to further protect the corporate network by using two firewalls instead of one.

For the home network, the first NAT would be between your ISP (Comcast, ATT, etc.) and you - your cable modem (or FIOS modem, 'the modem').  The ISP network IP address range is converted once.
If you go straight from your modem to a switch, you are probably good.

The second NAT happens when you put a router behind your cable modem.  Thus you end up traversing another IP address change.  Thus a second IP change.  The key here is that the public side of this second NAT is private behind the first firewall of the modem.

This looks like this:  - modem - - router -
The problem is that any modifications you make to your router get trapped in this container .
Not even port forwarding or DMZ will work properly here unless your cable modem supports uPNP or does not offer a firewall at all ( most give a firewall and default you into this first NAT ).

The things here, simply put your model into 'bridge' mode.
What this does is turn off this first NAT, and your router gets the public side IP from your ISP instead of your cable modem getting it.
The impact here is that for most of us, this means only ONE device can plug into the modem.  As you only get one IP of the ISP network.

If you cannot put your modem into bridge mode, put attach your Xbox directly to the modem (using a switch to go around your router).
I am always assuming that there is still some network protection here, at least a basic firewall.  As just about anything can be hacked these days.



I always want folks to understand how things work, how things function and to tie as much as what I uncover together in a cohesive way.  That is what I have attempted to do here.

I hope it helps some folks, or solves the problem for others.
It is very frustrating to bring a new game home ($80 - $100) only to have it not work (fail to load) at all.  From the number of passed customers on the web, I am not alone.

Tuesday, June 2, 2015

Linux VMs not getting IP with Hyper-V wireless external switches

For the past two days I have been building Ubuntu VMs on my laptop which runs Hyper-V.

I would install an Ubuntu VM, then try and update and discover that the VM has an IPv6 address but no IPv4 IP address.
So, off into the land of tweaking Ubuntu.  No go.

Next, my kids report router problems.  So I assume there is a correlation and I screw around with the router.  No change.

I delete and re-create virtual switches.  No Change.

After a bit of frustration and some calming attention to detail, I realize that the IPv6 address that my VM is getting is actually self generated, it was not getting it from my ISP (as I originally thought - since we do IPv6).

The one pattern is that; a virtual switch on the wired NIC always works with the VMs and the wireless doesn't.
The other pattern is that Windows VMs are just fine.  It is only Linux.

Now.  Since this is an external virtual switch that includes a wireless NIC a Network Bridge device is added.
 
I decided to poke around a bit.  I checked the properties of the Wi-Fi adapter (as it is common for power management to mess things up).  I discover that I cannot edit the driver properties of the Wi-Fi adapter due to the Network Bridge.

 
If I open the properties of the Network Bridge, I can then disable Power Management on the NIC. 
Come to find out tat was a waste of my time.  But hey, I had to try it.
 
But, wait a minute.  The bridge is dependent on the NIC.  ...  Network bindings pops into my head.
 
It used to be really easy to get into the bindings and totally mess things up.  Needless to say, it is not so intuitive any longer.
 
At the Network Connections press the ALT key, this reveals the file menu - select Advanced and then Advanced Settings.
 
What I notice is that the binding order is: Network Bridge, Wi-Fi, Ethernet

My thinking is; if Network Bridge is dependent on Wi-Fi, shouldn't it be after Wi-Fi?
(if you cluster or have clustered or have been around Windows as a server admin for a while you have probably messed with this before)
 
So I decided to give it a shot and move the Network bridge after Wi-Fi.
 
I then reboot for the change to take effect.
 
I then attach a VM to the virtual switch on my wireless NIC, cross my fingers, and power it on.
The VM boots right up, no hang at networking.  I logon, I type ifconfig and voila the VM has a proper network configuration.  I run 'sudo apt-get update' and all is glorious and good.
 
Just to fun, I build a Generation1 VM and install the pfsence router into it. 
That failed the auto configure test, but after reboot it came up just perfect (and it didn't prior).  And the latest version has the integration Components built-in and can use synthetic virtual NICs instead of Legacy - and even reports the IP address to the networking tab in Hyper-V Manager (I love that).
 
So much pain and consternation, for what now feels like a binding order bug.
I will update this is anything changes, but in the mean time: It works!
 
Now, why might Windows VMs work just fine? 
Because they keep trying to get an IP, they don't just try once at boot and then fail.  So that network stack can come live at any time, in any order (and generally does late in the boot sequence).

Wednesday, August 28, 2013

PowerShell to test if a network connection is up and on the domain

If you have noticed I have been spending a lot of time working with deployments, deploying, and scripting configurations.
In fact, I have spent nearly two years, off and on, working on this in various ways and permutations from Windows Azure VMRole (the now dead non-persistent one) to SCVMM Service Templates.
The thing that makes this type of scripting unique is that the scripts are executed within the OS of the VM, not externally from some manager that uses a PowerShell remoting session or the like.
This means that each script has no knowledge of anything beyond the boundaries of the OS where the script is running.
Now, I assume that many of you are aware of the Hyper-V Synthetic Nic, and that the Synthetic NIC driver comes to life later in the boot process (not in 2012 R2 generation 2 VMs, but that is different).
The problem is one of timing.  Your script could be running prior to your network being awake an functional.
Here is a little script that I use to test my domain joined machines prior to continuing when I have a need for domain connectivity (such as executing a command using a domain credential).

Do { $upTest = ( Get-NetConnectionProfile | where {$_.IPv4Connectivity -ne "NoTraffic"} ) } until( $upTest.NetworkCategory -eq "DomainAuthenticated" )
If you want to take this to the next level and identify the IP address and physical NIC (say you have multiple NICs and you need to bind to the IP of the domain NIC or the NIC itself in some configuration.

$mgmtNetProfile = Get-NetConnectionProfile | where {$_.NetworkCategory -eq "DomainAuthenticated" }  # Assuming only one NIC is domain joined.
$mgmtNetIpAddress = Get-NetIPAddress -InterfaceIndex $mgmtNetProfile.InterfaceIndex -AddressFamily IPv4

Tuesday, May 7, 2013

The IP the NIC on a particular domain or address space

I have this VM.  This VM has multiple interfaces.  One interface is _the_ management interface and it is the one that the VM used when it joined my domain.

I want to discover this NIC.  I then want its IP address so I can embed that into a configuration file.

The initial list of questions that I began with were:

  • Am I domain joined?  If so, what domain?
  • What network connection reports that domain?
  • What NIC is that?
  • What is the IPv4 address on that NIC?

I have discovered that there are multiple ways to handle this.

First of all, asking the questions; Am I joined and what domain am I joined to.

$env:USERDNSDOMAIN or Get-WmiObject -Class Win32_ComputerSystem | select domain

Both give you results, but different objects back.

BRIANEH.LOCAL (a string) vs.

domain                (a portion of a WMI object)                                                                                                
------

brianeh.local

Now, lets look at the question of what NIC is on what domain or connected to any domain.  If you only have one NIC connected to any domain (or that can resolve any domain) you can probably simplify this to:

Get-NetConnectionProfile | where {$_.NetworkCategory -eq "DomainAuthenticated" }

The alternate to that is to be more verbose (or precise if you like).

Get-NetConnectionProfile | where { $_.Name -eq $env:USERDNSDOMAIN }

What you get back is a Connection Profile object.  And to some this looks familiar, you know that status that you see on the network icon in your system tray?  The one that says if you have internet connectivity, or what DNS domain is discovered?  This is the information behind that.

I have two and they look like this:

Name             : Unidentified network
InterfaceAlias   : Ethernet 2
InterfaceIndex   : 13
NetworkCategory  : Public
IPv4Connectivity : LocalNetwork
IPv6Connectivity : LocalNetwork

Name             : brianeh.local
InterfaceAlias   : Ethernet
InterfaceIndex   : 12
NetworkCategory  : DomainAuthenticated
IPv4Connectivity : Internet
IPv6Connectivity : LocalNetwork

In the example I went straight to the selection of the desired one.

Now, how do I get to the IP you might wonder.  One more step.

Get-NetIPAddress -InterfaceIndex $netProfile.InterfaceIndex -AddressFamily IPv4

Again, I went straight to the IPv4 filter.  I used the Network Profile object captured as $netProfile and its InterfaceIndex property.  But, as you play with the Network IP Address object you will see that it can be selected multiple ways.

And if you only want the IP address itself, take that Network IP Address object and select only the IPv4Address property.

$netIpAddress.IPv4Address

And there you have it.  Now, off to building my configuration file…

Monday, November 14, 2011

Hyper-V WMI MAC address to a different format

Here is a simple one that I don’t want to lose.

In my previous post I had mentioned how to get the MAC address that is assigned to the vNIC of a VM.

Get the VM, then find the associated vNIC (and make sure the VM is running):

$vm = Get-WmiObject Msvm_ComputerSystem -Filter "ElementName='$vmName'" -Namespace "root\virtualization" -ComputerName $HyperVHost

$vnicEmulated = $vm.GetRelated("Msvm_EmulatedEthernetPort")

$vnicSynthetic = $vm.GetRelated("Msvm_SyntheticEthernetPort")

In the comments of my previous post Ben Armstrong made the following suggestion (which works kind of nice):

$vssd = $vm.getRelated("Msvm_VirtualSystemSettingData") | where {$_.SettingType -eq 3}
    $vmLegacyEthernet = $vssd.getRelated("Msvm_EmulatedEthernetPortSettingData")

The MAC address is the “PermanentAddress” property of the $vnic and the “Address” property of the $vmLegacyEthernet – so pay attention to the property you are fetching.

A quick way to pluck it out is just to loop through the collection like this:

foreach ($e in $vnic) {
    $mac = $e.PermanentAddress
}

This gives you a MAC address that looks like this:  00155D680A14

This is all fine and dandy if the reason you need the MAC can deal with that format.  And mine can’t.  I must have it defined with dashes.

I went searching for fancy REGEX ways to handle this and could not come up with anything.  But some nice validators for MAC addresses.  The best in the Splunk forums.

So, here is my less sophisticated but useful part:

$macDash = $mac.Substring(0,2) + "-" + $mac.Substring(2,2) + "-" + $mac.Substring(4,2) + "-" + $mac.Substring(6,2) + "-" + $mac.Substring(8,2) + "-" + $mac.Substring(10,2)

This gives me the following result:  00-15-5D-68-0A-14

Just add that into the foreach loop above and move on.

Friday, April 8, 2011

Unable to ping Server 2008

I have blogged about this before and I will again as folks fall trap to this all of the time.

By default, ping response is disabled beginning with Server 2008 and continuing into newer revisions of Windows Server. 

FYI - With Server Core, ping is silent regardless of the state of the firewall.

Here is a great tip from Ben of the Hyper-V team:

The first thing I would do is to check the firewall configuration in your guest operating system, and make sure that it allows ICMP.

Here is a great article that steps you through how to do this:

“Nobody can ping my computer”

http://technet.microsoft.com/en-us/library/cc749323(WS.10).aspx

Wednesday, March 30, 2011

VM to VM network communication in Azure

VM to VM communication for VMs running on Windows Azure is not just up and straightforward and wide open like it is when you run a VM on a hypervisor in the enterprise.

By design, Role Instances (these are VMs) in Azure have an outer security wrapper around them.  This establishes that final trust boundary for the VM container.  You can see this when you use VM Role.  Because beyond your Windows Advanced Firewall you have an additional level of block to your network connectivity.

This is where you must define your endpoints (and you must know your application and the required ports very well) within your Azure Service.

In the properties of any Azure Role is the Endpoints property.  these are little port specific pin-pricks in the Azure VM runtime container specific to network traffic.  You poke a hole, you allow incoming traffic on that particular port.  Don’t get hung up on the endpoint term – it is very much a developer type term for “something that I can connect to, or talk to.”

WI_InputEndpoint

The Name is whatever you (or your developer) calls it.  the type is Input or Internal.  The Protocol is HTTP, TCP, or HTTPS.  Public ports apply to Input endpoints.  Private port applies to both Input and Internal endpoints. 

An Input endpoint is where your Azure Service is exposed to the wild world.  It is the public port that anyone can touch. 

An Internal endpoint is internal to your Azure Service and is where any Role Instance can touch another Role Instance over the network.  Here you set the Private Port number.

This opens a port in the envelope that allows incoming traffic to the role instance where this is defined.  And in the case of a public port, it maps that through a load balancer to a public IP.  Otherwise, your internal traffic is over an internal IP subnet (not a public range).

I envision this to look a little like this:

GetStart_Trust

Now, the tricky part comes next.  The setup of your application in Azure can be exceedingly complex because of one rule that Azure has; your OS images are prepared with sysprep.  This is so that any Role can exists as multiple instances and be totally unique.

Under the hood, there are traditional Windows Setup and Deployment tricks happening all over the place.  Unattended setup, pre-tasks, post-tasks, user elevation, injection of applications, certificate installation – all kinds of things are happening or can happen when a Role Instance is provisioned.  This is all related to the Azure way of doing things.

Think about this.  You have an enterprise application.  Does it support being sysprep’d?  Many don’t.

Also, how does your application behave without any type of name resolution?

And, how do you discover your other role instances without connecting to and logging into each one?

For Web and Worker roles Microsoft wants you to use the AppFabric or Azure Storage (blobs, queues, tables) to enable this communication.  But what if re-writing is not an option?

Tuesday, July 6, 2010

Ping is dead on Windows Server stop using it

Long live Ping!

For many years we have relied on Ping as a quick and easy measure of a server being ‘alive’ or not.

I have been stating in the TechNet forums since the release of Server 2008 that we have to get off the Ping train.  It is no longer a real measure.  We cannot expect it to be open and on.

Just today, I am installing a new test environment with Server 2008 R2 (all Enterprise edition, all built from scratch, all domain joined).

I began installing my applications, all fine, until I try to connect to my SQL database server (it is a VM of course).  What is the problem? I had added a Firewall rule.

Without even thinking, I pull out Ping.  hmm.. no response.  <All machines are domain joined, I expect the domain firewall rule to let me ping…>

hmm.. again, no response.  I check the domain controller, I check DNS, I run out of ideas.  So I go into the firewall rules.  One by one I disable to firewall while I have Ping running (just to make sure that my traffic is being detected as Domain traffic).

I began with Public, then Private, then Domain.  Well, yep, the traffic is being correctly detected as Domain traffic and Ping is blocked by default!

Just goes to show you that as an operating system gets secured tighter and tighter, that some quick and easy tools fade into the background.  If you want Ping then set a GP firewall exclusion for Ping, or simply move on to using something different…and focus on the fact that Windows Firewall actually works really well.

Tuesday, December 22, 2009

Hypervisor virtualization basics a visual representation

For all of you that want a place to point folks to describe the basics of virtualization, I have put together a few videos to describe the hypervisor, CPU, and network concepts in a visual way.

The intent is to give a quick amount of conceptual information to those folks that suddenly are dealing with VMs, but might not have the experience to fully understand what they are looking at.

Hypervisor Basics:

The basics of what a full (type 1) hypervisor is.

 

The hypervisor pool of resources – the CPU:

The basics of CPU scheduling. It is far more complex than this and there are many methods.  It gets really messy when hyper-threading is introduced.

More CPU scheduling details are over at the Xen.org site (the Xen folks are more open in discussing the gritty details of all this):  http://wiki.xen.org/xenwiki/CreditScheduler

The hypervisor pool of resource – the network:

This is about virtual networks / virtual switches / bridging.  The concepts, as each vendor implementation offers different features.

Tuesday, July 21, 2009

More about Chimney and TCPOffload in Hyper-V

Here are some definitions that help to clarify the TCPOffload and Chimney thing really well.

I have Don from the Hyper-V networking team to thank for the detail.  Being a hard-core networking guy he knows his packets.

Chimney, also known as TCP Chimney Offload, is the offloading of all IP and TCP processing to the NIC.  This means the NIC receives the packet, processes the headers, generates the ACKs, and keeps all the state.  On the outbound side it receives a block of data from the app, packetizes it, generates the headers, generates the IP layer, and ships it.  Chimney is available on some Broadcom NICs.


Checksum offload is the offloading to the NIC the responsibility for generating the header checksums (outbound side) and verifying the header checksums (inbound side).  No header processing is done other than the checksum processing.  No state is maintained in the NIC.  Nearly all server class NICs support checksum offload.


Large Send Offload (LSO and LSOv2) is the offloading, on the send side, of the packetization and header generation.  The hardware takes a large data block and, using state information from the stack, generates appropriate size data packets (including the headers).  The state is kept in the stack.  LSO and LSO v2 are different versions of this feature.  LSOv2 is supported in R2.


In summary: if you are using Chimney you receive the benefits of the other two.  Disabling Chimney does not disable either LSO/LSOv2 or checksum offload.

Tuesday, June 24, 2008

DMZ isolation of VMs with Hyper-V

I recently posted this in the TechNet forums and thoguht that it needed a bit longer life.

Mostly, this is about understanding virtual machine networking under a hypervisor (ESX, XenSERver, Hyper-V, etc.) and that they all basically work the same.

The simplest way to accomplish isolating VM traffic into a DMZ is to do a form of physical network isolation.

Your Host has two physical NICs.

Connect NIC 1 to the management network. Connect NIC 2 to the DMZ network.

Create two External Virtual Network Switches - one per physical NIC and name them appropriately.

Connect your VMs only to the DMZ virtual network switch (using the VM settings).

Login to your Hyper-V Host console - check the network connections. You most likely have a Virtual Network Adapter (possibly two).
You may end up with one for each virtual network switch. If this is the case, open the properties of the Virtual Network Adapter that is attached to the DMZ virtual Network switch and remove (uncheck) all protocols - this will prevent any traffic from the Host into the DMZ and vice versa.

The alternative is VLAN IDs.

Now..backing up.

Looking at network connections within the Hyper-V console can be confusing.

When you first install the Hyper-V role you have Physical NICs and properties just like you know and love from any Windows server. As soon as you create an external virtual network switch the physical NIC only has the virtual network protocol bound to it (it is no longer 'owned' by the Host, but instead by the Hypervisor) - leave this NIC alone.

If your Host was using this physical NIC it is now given a new virtual network adapter that is attached to the virtual network switch which is in turn using the physical NIC - therefore keeping the host on the same physical network.

The part that makes this confusing is that we CAN look at the network connections, and we CAN play with things just like we have with Windows Server for years. However, now we have to realize that the architecture is different - a mix of what we knew, a dash of what we saw with Virtual Server on Server 2003, and parts that are totally new becuase we have a hypervisor and a virtualization stack.

Thursday, May 15, 2008

Hyper-V + TCPOffloading = poor network performance?

* Update * - Here I use the term TCP Offload.  The actual feature is properly termed TCP Task Offload.  TCP Offload is better referred to as TCP Chimney and that should never be turned off.


Here are some tips that I have passed on regarding Hyper-V and TCPOffloading.
http://forums.microsoft.com/TechNet/ShowPost.aspx?PostID=3107983&SiteID=17

In theory TCPOffloading should always make things better. It takes work away from your server OS, thus everything should work better. Right?

As a former System Administrator, I have been disabling TCPOffloading on Windows Servers for years. Always to fix network performance issues. This works for SQL servers, Terminal Servers, and now Hyper-V servers.

Why? I honestly have not gotten deep enough into the issue to fully understand why, I just know that if your network throughput is pathetic on a Windows Server that is running some application or role, try this trick.

(as always take caution, this might work, it might have no effect - I take no risk from you)

In this case, focus on the NIC driver on the base network connection. (not the virtual one that your parent partition uses, but the Physical NIC properties (the NIC with only the Microsoft Virtual Network Swith Protocol enabled))

Make sure it is either the WS08 provided driver, or the latest from the manufacturer (it is usually always safer (read this as: overall more stable) to use the one that ships with Windows - unless you need a feature provided by a different driver)

Try turning off TCP offloading - as this can cause strange behavior on some networks.

Also, with Hyper-V if you are using a manufacturer provided driver and you want to do fancy teaming and stuff like that at the driver level - please make sure the driver has been updated for Hyper-V, or else you will get weird results here as well.

Don't use NIC teaming, or NIC management software unless the manufacturer says it is Hyper-V role happy (not just WS08 happy).

Some NIC teaming management software needs special control of the hardware and it might not work after Hyper-V is installed, since the physical hardware is now shared.

Wednesday, April 2, 2008

Networking under Hyper-V (who moved my network settings?)

There is a lot of discussion around networking and network changes on a Windows Server 2008 server after the Hyper-V role is installed.

Part of this issue goes back to what happens (what is done) to your WS08 server when the Hyper-V role is installed.

As I had mentioned in a previous post when the Hyper-V role is installed the WS08 server is fundamentally changed. As an administrator you login to a console and you see WS08 - it looks like nothing changed, however it has.

Ben Armstrong has a quick posting here that gives some basics.

Lets take a more architectural look at the server and what has happened / is happening..

During the process of adding the Hyper-V role the WS08 installation was turned into a virtual machine itself and it is running on top of the hypervisor as the parent partition. Unlike other hypervisors where you see a Linux based console, in this case you see your WS08 server - a nice, friendly GUI interface that really does not look any different than it did before.

Now, what does this have to do with the networking? It has to do with modifications that were made to the network interfaces of the WS08 server when the Hyper-V role was installed.

Quite honestly, the modifications that are made are no different than what happens when modifying any other Windows server in a similar way.

Most likely, if you set a manual IP address (any manual settings) they were lost.

If you look at Network Connections in the WS08 server you notice new Virtual Network Adapters and your original network connections were changed.

Noting back, your WS08 server was turned into a VM (its hardware was changed) - the original network connection (which you can still see) was turned into a virtual switch (the WS08 server no longer owns that NIC).

Your WS08 server (WS08 parent partition - that is what it is now) was given new virtual network card(s). And as with adding any new NIC to any Windows server it gets the default settings (DHCP for example).

Now, there is also talk about performance (the parent partition performance is terrible, but a VM runs great - or the other way around).

Now we have to begin looking at the NIC driver and driver configuration.

First of all, good old TCP Offloading, long a performance issue in many Windows environments might need to be turned off. Mind you, this issue seems to be environment specific.

The other is the NIC driver itself. You are pretty safe using the included WS08 drivers.

Some troubleshooting questions:
Did you install a non-Windows delivered driver?
Did you install a teaming driver?
Did you configure teaming?
Was there management software installed with the teaming driver or was only the driver installed? (some experience has shown that the driver itself might be fine but the management software causes problems - as it is trying to monitor a NIC that the parent partition no longer owns)

My goal with this post was to help an administrator understand what is going on, and with that where to look to solve his/her problems.