TCP Checksum Offload is not equal to TCP Task Offload

This is an issue that lately I have been answering a lot in the Hyper-V TechNet forum. 

Folks find a link that refers to disabling Checksum Offloading or TCP Offload  to help with strange networking behavior with an application server (Remote Desktop Services, XenApp, Exchange, SQL, SharePoint, etc.).  So they disable “TCP Checksum Offload” using netsh on the Hyper-V Server.  The end result is that the problem is not resolved, so they come to the forum.

In actuality they did not disable the correct feature.  So here is my third attempt to try and straighten this out.

In this particular scenario the features (yes, multiple with some NIC drivers) are referred to as TCP Task Offload.  These are things such as TCP Checksum Offload, Large Send Offload, etc.  And they are properties of the NIC driver – it is not a single server level setting.

There is a really ancient Microsoft KB article that talks about this.  You want to use Method 3 in this article:  http://support.microsoft.com/kb/888750

I am going to paraphrase it here:

Method 3
If you do not want to disable TCP segmentation offloading on the whole system, and you want to only disable TCP segmentation offloading on the network adapters that Virtual Server 2005 guests use, you must not add the DisableTaskOffload registry entry that is described in Method 2. Instead, you can disable the task offload properties on the Advanced tab of the Properties dialog box of the network adapter.

Warning When you disable the task offload properties, guest virtual machines that are attached to the same virtual network may temporarily disconnect from the virtual network.

To disable the task offload properties, follow these steps:

1. Click Start, click Run, type ncpa.cpl, and then click OK.
2. Right-click your network adapter, and then click Properties.
3. Click the General tab, and then click Configure.
4. Click the Advanced tab.
5. In the Property box, click the Offload TCP Segmentation property.
6. In the Value list, click Off, and then click OK.
7. If you also have the following task offload properties in the Property box, you must repeat step 5 to step 6 to disable these properties:
• Offload Receive IP Checksum
• Offload Receive TCP Checksum
• Offload Transmit IP Checksum
• Offload Transmit TCP Checksum
  

If you have Server Core the following might help:

http://social.technet.microsoft.com/forums/en-US/winservercore/thread/d0c55df9-a27c-4876-bc5a-8ac7f1b46462/

“There are some settings for TCP/IP, go to "netsh interface tcp" and then run "set global" and you'll see all the options for some of the advanced TCP/IP configuration. One of those may help.”

The magical registry settings are documented in this MSDN article: Using Registry Values to Enable and Disable Task Offloading:   http://msdn.microsoft.com/en-us/library/ff571012.aspx

VM to VM network communication in Azure

VM to VM communication for VMs running on Windows Azure is not just up and straightforward and wide open like it is when you run a VM on a hypervisor in the enterprise.

By design, Role Instances (these are VMs) in Azure have an outer security wrapper around them.  This establishes that final trust boundary for the VM container.  You can see this when you use VM Role.  Because beyond your Windows Advanced Firewall you have an additional level of block to your network connectivity.

This is where you must define your endpoints (and you must know your application and the required ports very well) within your Azure Service.

In the properties of any Azure Role is the Endpoints property.  these are little port specific pin-pricks in the Azure VM runtime container specific to network traffic.  You poke a hole, you allow incoming traffic on that particular port.  Don’t get hung up on the endpoint term – it is very much a developer type term for “something that I can connect to, or talk to.”

The Name is whatever you (or your developer) calls it.  the type is Input or Internal.  The Protocol is HTTP, TCP, or HTTPS.  Public ports apply to Input endpoints.  Private port applies to both Input and Internal endpoints. 

An Input endpoint is where your Azure Service is exposed to the wild world.  It is the public port that anyone can touch. 

An Internal endpoint is internal to your Azure Service and is where any Role Instance can touch another Role Instance over the network.  Here you set the Private Port number.

This opens a port in the envelope that allows incoming traffic to the role instance where this is defined.  And in the case of a public port, it maps that through a load balancer to a public IP.  Otherwise, your internal traffic is over an internal IP subnet (not a public range).

I envision this to look a little like this:

Now, the tricky part comes next.  The setup of your application in Azure can be exceedingly complex because of one rule that Azure has; your OS images are prepared with sysprep.  This is so that any Role can exists as multiple instances and be totally unique.

Under the hood, there are traditional Windows Setup and Deployment tricks happening all over the place.  Unattended setup, pre-tasks, post-tasks, user elevation, injection of applications, certificate installation – all kinds of things are happening or can happen when a Role Instance is provisioned.  This is all related to the Azure way of doing things.

Think about this.  You have an enterprise application.  Does it support being sysprep’d?  Many don’t.

Also, how does your application behave without any type of name resolution?

And, how do you discover your other role instances without connecting to and logging into each one?

For Web and Worker roles Microsoft wants you to use the AppFabric or Azure Storage (blobs, queues, tables) to enable this communication.  But what if re-writing is not an option?

Random Wait PowerShell Script

During my work with Azure I have run into cases where the provisioning of service instances causes flooding of some other component – such as the backend Azure SQL database as my instances register themselves into a configuration database.

To get around this very traditional “black hole” type of problem I have a very simple PowerShell script that I run in the proper sequence as a First Logon Command as my instances complete mini-setup.  (The OS is prepared with sysprep so everything has to go through mini-setup).

The command line in my unattend.xml looks like this:

<FirstLogonCommands>
  <SynchronousCommand wcm:action="add">
    <Order>1</Order>
    <CommandLine>%WINDIR%\System32\WindowsPowerShell\v1.0\PowerShell.exe -command set-executionpolicy remotesigned -force >> C:\Users\Public\Documents\setExecution.log</CommandLine>
    <Description>Set the ExecutionPolicy to RemoteSigned for the script to run</Description>
  </SynchronousCommand>
  <SynchronousCommand wcm:action="add">
    <Order>2</Order>
    <CommandLine>%WINDIR%\System32\WindowsPowerShell\v1.0\PowerShell.exe C:\Tanzanite\RandomWait.ps1 >> C:\Users\Public\Documents\RandomWait.log</CommandLine>
    <Description>A random wait time to prevent storming and corrupting the database</Description>
  </SynchronousCommand>
  </FirstLogonCommands>

You will notice that the first thing I have to do is to issue a command to set the ExecutionPolicy.  After mini-setup the ExecutionPolicy is reset to Restricted and my script will not run.

BTW – FirstLogonCommands are executed when an Administrator logs on to the machine for the first time.  So, prior to this I am using AutoAdminLogon to set the administrator to logon and then these scripts execute.

Now, the PowerShell script.  (RandomWait.ps1)

<#
.SYNOPSIS
    This is a frivolous script, strictly to be called to introduce a random wait period
    within a five minute window, based on a random number of seconds.
.DESCRIPTION
    This script is designed to facilitate VM instance application setup in the Azure Environment.
    To prevent database corruption during deployment this random wait
    is introduced after provisioning and called as an UNATTEND.xml FirstLogonCommand
    during mini-setup.
.LEGAL
    SCRIPT PROVIDED "AS IS" WITH NO WARRANTIES OR GUARANTEES OF ANY KIND, INCLUDING BUT NOT LIMITED TO
    MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE.  ALL RISKS OF DAMAGE REMAINS WITH THE USER, EVEN IF THE AUTHOR,
    SUPPLIER OR DISTRIBUTOR HAS BEEN ADVISED OF THE POSSIBILITY OF ANY SUCH DAMAGE.  IF YOUR STATE DOES NOT PERMIT THE COMPLETE
    LIMITATION OF LIABILITY, THEN DELETE THIS FILE SINCE YOU ARE NOW PROHIBITED TO HAVE IT.  TEST ON NON-PRODUCTION SERVERS.
.AUTHOR
    Brian Ehlert, Citrix Labs, Redmond, WA, USA
.REFERENCES
    Thank you TechNet. For examples.
#>#

# Add the Service Runtime snap-in to the standard Windows PowerShell command shell.
add-pssnapin microsoft.windowsazure.serviceruntime

# Take the VM Instance offline with Azure
Set-RoleInstanceStatus -Busy

# Get the random number generator object so as not to require PoSh v2.0
# PoSh v2 introduces Get-Random
$randomNo = new-object system.random

# Generate a random number between 1 and 300
# using seconds instead of minutes as chance for it to be not equal to other instances is higher.
$number = $randomNo.Next(1, 300)

Start-Sleep -Seconds $number

SCVMM 2012 XenServer demonstration tricks

This is a little trick that I picked up from an unnamed MSFT person.

Say that you want to have a portable demonstration environment for SCVMM 2012 to show off all of the bells and whistles.  You have a limited amount of hardware, say two servers. (maybe even one).

You can install Hyper-V.  And you can install SCVMM into a VM.  That is all fine and dandy.

You can then install Hyper-V Server into a VM as well.  And this looks good until you try to do something that requires the VM within the virtual Hyper-V to power on.  Then it falls apart.

Well…  you can mix your hypervisors up.

You could install a XenServer within a Hyper-V VM and even create a pool using two or more.  They will join and become a Resource Pool.

The key is that you give the XenServer VM a Legacy Network Adapter for installation to proceed.

You can then deploy VMs to the XenServers using SCVMM 2012 (or XenCenter if you like).  And the VMs will boot.  (yes, all disclaimers apply – a hypervisor in a VM is not a place where you expect anything to be fast, but it can show that something works).

There is also a sneak here as well.  Windows VMs will not boot – the reason is that a Windows VM requires an HVM type VM, which requires that the hypervisor in the VM must own the hardware – and it doesn’t.  Linux VMs will boot just fine.  As they can exist as PV VM’s in XenServer or as HVM type VMs.  It is the PV VMs that will boot.

Now, you can always use Hyper-V and install that into a VM running on Hyper-V.  I know folks that do that.  And these can be managed as well – but none of the VMs can be powered on.  So if you need to test SCVMM 2012 and you need to show VM power operations, you need to use Linux VMs running in a virtual XenServer.

SCVMM 2012 beta with XenServer support

The SCVMM 2012 beta was announced at MMS.  Go pick it up here:  http://www.microsoft.com/downloads/en/details.aspx?FamilyID=e0fbb298-8f02-47e7-88be-0614bc44ee32

Now, I will warn you.  This is a big update. Not just some little incremental update.  It includes some new ways of thinking about your data center.  And it includes a lot of “cloud” type words and ways of thinking.

I consider this update a paradigm shift in the VMM world.  A big change in thinking, automation, abstraction, packaging, and delivering.

It also includes support for managing XenServer.  This is a feature that is near and dear to me.  There is a supplimental pack that is required for your XenServers which can be obtained from here: http://www.citrix.com/xenserver/microsoft-beta

This gives the ability to deploy too and manage XenServer.  You still need to setup your environment using xe or XenCenter (no different than Hyper-V) – but you can control and manage VMs with SCVMM.

Other MSFT folks have mentioned the other big paradigm changes, so I won’t cover them here.  http://blogs.technet.com/b/scvmm/archive/2011/03/22/system-center-virtual-machine-2012-beta-available-now.aspx

Azure VM Role and Application Setup

I have never considered myself an unattended setup type of person.  Mainly because for most applications the process has always been too painful or the potential for danger is too high.

Needless to say, I have been spending a great deal of time with Windows Azure and the VM Role and a few applications.

What have I learned from this?

• Not all command line utilities can be invoked properly using ‘&’ or ‘cmd /c’ from PowerShell.
• I have a couple applications that just don’t execute into a command windows from powershell.  I have worked around this by determining my variables and then writing a .CMD file using Out-File.
• I can then use ‘&’ to run the .CMD file and it works perfect.
• Not all unattended installers work as documented.
• ‘nuf said
• FirstLogonCommands in the unattend.xml file is highly useful.
• You can execute PowerShell, you can execute individual commands, you can install server roles and features, lots of stuff.
• Software in Azure doesn’t always debug the same as on Hyper-V
• At the very least network communication is different.
• The Azure VM runtime container is different and is managed differently, to say the least.

I will be posting some PowerShell snippets that I have found useful and we will undo an Unattend.xml file as well.

All in due time.

Read this before XenApp6 with 2008 R2 SP1

I ran into this one myself, and it was a bit troublesome to figure out.

If you currently have XenApp6 and you have a real urge to install SP1 for Server 2008 R2 be sure to read this KB article and get the hotfix lined up – before you install SP1.

http://support.citrix.com/article/CTX126711