Testing and Checking in the cloud service era

Services.  DevOps. Cloud Services. Rapid prototyping. Start-up methodologies. Agile development.

These are all terms that we hear in the software business.  And they generally all point to what some folks consider a reduction in testing.  And what many customers consider a reduction in overall quality.

If you follow some of the big names in the software testing world ( Michael Bolton, Rex Black, James Bach, and others ) one theme that resonates with all of this is testing vs checking.

And in my experience what I see happening in this cloud service era and the rise of DevOps is that there has been a shift away from 'testing' and a greater focus on 'checking'. 
Some say that this makes the developer more accountable instead of them blaming test for missing it.  I don't argue that quality was always the developers responsibility anyway.

In my words, checking is more like unit testing.  Did this thing respond in the way it was designed to?  Everything is positive, everything is looking for the intended desired outcome.

Where testing is checking plus looking at and driving appropriate failures.  Not only did this thing do the positive action, but when I send a negative action - what happens.  Did it fall over, did it respond with a proper error, did something totally unexpected happen.  And then there are other studies; load, scaling, fuzzing, chaos monkey, etc.

In fact, I would argue that in this cloud service world, this shift to a greater emphasis on checking is actually a bad thing.  I tis good for development in that it gets more code out the door, and in theory more features and quicker fixes.  However, it ends up making for very fragile applications, and I know of few cloud applications that don't have a high number of dependencies on other cloudy services.

Now, why does this matter?  Because, customers have an emotional connection to your product.  Not a factual relationship with it.

Any small issue that blows up into a large issue, or a planned two hour outage that becomes a 4 to 8 hour outage impacts the feelings of the customer in regards to your service.  This impacts their perception of your quality.  As the customer sees quality as a value judgment.  No different than excellent service at a restaurant.  The more expensive the restaurant, the better everything must be.

This gets messy in the cloud service era.  Because of pricing competition.  The model is closer to that of a bank.  If a customer only consumes one of your services, it is easy to switch.  So you upsell them with more and more services and possibly take a loss in doing it.  This generates a type of lock-in as it becomes more and more difficult for a customer to leave and go somewhere else.  The tipping point.

Anyone who has been involved in IT purchasing decisions knows that software is brought into the enterprise through some corporate division.  Then it gets handed to IT to evaluate.  And in very rare cases, IT actually gets to give feedback or get the corporate division to consider alternate options.
The other side to this is that IT is the one doing the evaluation and choice.  When this is the case the judgment is all about the getting started experience - if I have to crack a manual to get started, something is not right.

Again, checking would focus on the binary functions of the software where testing should be looking at the overall experience.
Just something to keep in mind in this rapid software era.

Synchronizing PowerShell script execution on multiple machines

I originally thought to call this “synchronizing watches” but since I am not changing system time I thought it might be misleading.  This post is all about synchronizing script execution on two different machines.

Some folks would use a central client and workflows (PowerShell v3) or PowerShell remoting to execute a script on remote machines at the same time.  Possibly even executing each one as a job so they happen at the same time. 

It also allows me to launch the script on server A and then move to the other side of the lab to launch its script and then begin the repro (or switch the monitor on the KVM).  I know that as long as they share a time service I can be confident in when the tracing began on the remote machine.

In my case I want to execute the script on each machine at the console.  My reasoning is that; the machines are not in the same domain making security messy, or I don’t want to modify firewall rules, or I am capturing network traffic and just don’t want the extra noise to filter through.

Here is my scenario: 

• I have multiple servers (a large, distributed application)
• Different things are happening on each server
• I want to enable a network trace at the same time (to have nicely correlated logs)
• The systems already have a common time source (the OS handles that after all)
• I need to manually trigger the reproduction of the bug after the tracing starts, or I need to trigger some event prior to the tracing beginning
• I only have two hands and two feet

Here is the synchronization snip at the beginning of my script:

Do {
    Start-Sleep ( 60 - (Get-Date -Format ss) )
    
    [string]$nowMinute = Get-Date -Format mm
} until ( $nowMinute[($nowMinute.Length - 1)] -eq "0" -or $nowMinute[($nowMinute.Length - 1)] -eq "5" )

The first thing that I do I describe as squaring up.  I invoke a Start-Sleep but I want to sleep until the end of the current minute, no longer.

Start-Sleep ( 60 - (Get-Date -Format ss) )

You could do this as two lines as well (or three if you like):

$secToZero = 60 - (Get-Date -Format ss)

Start-Sleep $secToZero

Then I get the current minute and evaluate it to the nearest 5 minute block.

I chose the 5 minute mark because it is easy to evaluate to (if the number ends in 0 or 5 it divides by 5).  I cast the returned integer to a string as well, so I can get that last digit.

[string]$nowMinute = Get-Date -Format mm

*Notice that if you use “MM” instead of “mm” you get the Month.

I do my math evaluation at the end of the Until.  Notice the –or between the 0 and 5 after the –eq.  That is important to force the evaluation against both numbers.

Here is where personal preference sets in.  I dreamed up three different ways to handle the evaluation in the Until block.

The first is a very literal evaluation of the second digit in the Seconds to see if it equals “0” or “5”.  the entire last line looks like this:

} until ( $nowMinute[($nowMinute.Length - 1)] -eq "0" -or $nowMinute[($nowMinute.Length - 1)] -eq "5" )

It is the ($nowMinute.Length –1) that puts us at position 1 ( position begins counting at 0) to perform the evaluation.  The position indicator for the array is the square brackets [].

Another way to handle this is through the PowerShell way to handle a Regular Expression (or a pattern match – a nice article on that is here.)

Based on that article I can keep my –or and simply indicate the second digit this way:

} until ( $nowMinute -match ".0" -or $nowMinute -match ".5" )

Or, I can do it more like a regular expression and state in a shorter line that the second digit can match either 0 or 5.  That looks like this:

} until ( $nowMinute -match ".[0,5]" )

The end result is all the same.  Much of the technique is all about personal preference.

Works on MY cloud

For the current shift to the ‘cloud’ I have found it necessary to update the classic “works on my computer” badge.  (I have been threatening to do this for a long time)

Many of us in the software business are very familiar with the classic response to a bug of:  “No Repro” or “Not Reproducible”

There are many reasons for these types of answers, but the most common is that I am not the developer and I am not running the software on his machines or in the same way that she did.  This resulted in the classic “works on my machine”:

For the cloud era these applications are generally part of a large distributed system and that is what we are testing – many moving parts, many roles, many different systems.  Errors happen all over the place.  And everything is now a cloud.

So, my proposal has been to update the classic badge of jest to the cloud era and the “Works on my cloud” is born:

James Whittaker is back at Microsoft

For those of you not in the software testing circles this might not mean much to you.

James Whittaker is one of the pre-eminent folks in the field of software testing. 

It was sad a couple years ago when he suddenly left Microsoft for Google.  And interestingly enough he is back at Microsoft.

Most test folks are outspoken and critical.  Maybe that is what makes a good tester(?)  We call it as it is, trying to be accurate and factual.

There was one classic blog article that James wrote that I always loved and still see the symptoms of all over the world of software:  http://blogs.msdn.com/b/james_whittaker/archive/2008/08/11/if-microsoft-is-so-good-at-testing-why-does-your-software-suck.aspx

The title speaks for itself.

If you are interested, James has a new blog and he opens right up with a post critical of Google:  http://blogs.msdn.com/b/jw_on_tech/archive/2012/03/13/why-i-left-google.aspx

If you didn’t notice his blog names; back in the day is was JW_On_Test and today it is JW_On_Tech.

PowerShell DateTime to CIM_DateTime

Obviously no one that is using PowerShell is using WSMAN against a remote Linux system.  Everything assumes WMI, simple enough.

Use WSMAN against Linux and you enter into a insane land of XML and properly formatting your XML.

Take for example the simple act to send an XML string that queries a time period.

In PowerShell you type Get-Date and you get a nice, human friendly value back: Wednesday, May 05, 2010 10:24:14 AM

Now, try to send that to a CIM provider of any type (in my case a CIM provider that sits behind a WSMAN interface) and you immediately get a value invalid error.

off to Bing-land.. searching, searching, searching – absolutely nothing.  Wait, there are a couple useful things…

on MSDN the Scripting API Objects, the SWbemDateTime Object.  The what?  you say.  Isn’t it obvious? (I didn’t think so).

Here is the kicker, the CIM_DateTime format.  It expects this really strange format that looks like this:  yyyymmddHHMMSS.mmmmmmsUUU

So how do I take this:  Wednesday, May 05, 2010 10:24:14 AM and turn it into this: 20100505102415.000000-420

I have to play with objects in PowerShell, here is my script part:

$startTimeWindow = ((Get-Date) - $9Minutes)
$objScriptTime = New-Object -ComObject WbemScripting.SWbemDateTime
$objScriptTime.SetVarDate($startTimeWindow)
$startTime = $objScriptTime.Value

I first set my time window to begin 9 minutes before ‘now’.  I then create a SWbemDateTime object from the Wbem.Scripting class.  I then take the start of my time window and set this friendly formatted time to the object.  Then I retrieve the value of the object and I have a CIM_DateTime to send off to my Linux system CIM interface (through WSMAN).

Creating a Windows 7 or Server 2008 R2 image for VM deployment

Creating a Windows 7 image for deployment to a virtual machine is not as straightforward as you might think. If you simply perform a standard installation, image that to a WIM with ImageX and then attempt to deploy that image, you will be left with a system that requires that the boot manager be recreated.

Just in case anyone is confused as to why the boot partition is there, it's so that you can use bitlocker to encrypt your system drive on the fly.  Bitlocker needs the unencrypted boot partition to work.

If you create a VM and insert your Windows 7 media and proceed to click Next through the entire installation wizard you will end up with a system that is actually installed on two partitions and not deployed back from WIM without requiring repair.

One tool that can be used is the Wim2Vhd tool. This is useful if you meet the client requirements and have a way to import the resulting VHD into a new VM. It performs the required repair for you.

However, I would like to avoid the situation of needing to perform the BCDBoot repair all together.

Say that you want to use ImageX (from the Windows Automated Installation Kit ) to image the Windows 7 installation to a WIM and then deploy that Windows 7 image to other virtual systems. I do not want to repair each and every time – I want to build the reference system in a way that this is not required.

The key is to prevent the default Windows 7 installation behavior of creating a System Reserved partition (which contains the necessary boot loader files) and a separate partition where the OS is installed (where there are no boot loader files).

Method One – pre-partition the virtual disk:

This is the most direct approach; in the end nothing looks unusual.

1. Create the VM from a template

2. Boot to WinPE (or mount the disk to another (helper) VM)

(An alternative to this is to boot using the installation media.  At the "Install Windows" screen (the first screen of the installer) type Shift+F10 to open a command prompt)

3. Run diskpart

a. List disk

b. Select the disk

c. Create a primary partition (you must use the entire volume)

d. Make the partition Active

e. Format the volume

f. Exit

4. Power off the VM ( or detach the virtual disk from the helper vm)

5. Attach the Windows 7 installation ISO

6. Boot to the ISO

7. Begin the installation wizard

8. Select a “Custom” installation

9. At the “Where do you want to install Windows” screen accept the volume that was previously formatted

10. Complete the wizard

11. Apply all installations and patches to your VM

12. Prepare your VM with sysprep

13. Boot the VM to WinPE

14. Create your VM image using ImageX.

Method Two – confuse the installation wizard:

I could not think of a better title for this, because really what you are doing is messing with the options allowed in the installation wizard until you get the behavior you want.

1. Create the VM from a template

2. Attach the Windows 7 installation ISO

3. Boot to the ISO

4. Begin the installation wizard

5. Select a “Custom” installation

6. At the “Where do you want to install Windows” screen

a. Select the virtual disk

b. Select “Drive options (advanced)”

c. Select “New”

d. The default should be the entire volume – Select “Apply”

e. Acknowledge the warning

f. Select and Delete Partition 2

g. Acknowledge the warning

h. Select and Extend the System Reserved partition

i. The default should be the entire volume – Select “Apply”

j. Acknowledge the warning

k. Select Next

7. Complete the wizard

8. Apply all installations and patches to your VM

9. Prepare your VM with sysprep

10. Boot the VM to WinPE

11. Create your VM image using ImageX.

The primary difference between the results of the two methods is:

· Method Two results in a deployed image where the C: volume name is “System Reserved”

Method One:	Method Two:

BOOTMGR is missing - Repairing a Windows 7 or 2008 R2 image after VM deployment

Deploying a Windows 7 image to a virtual machine is not as straightforward as you might think.
The primary issue resides in how the reference Windows 7 system is installed.
If you create a VM and insert your Windows 7 media and proceed to click Next through the entire installation wizard, you will have a working Windows 7 installation, that can be templated, it can be copied, you can do just about anything you like to it – as long as all copies involve the entire virtual disk.
Say that you want to use ImageX to image the Windows 7 installation and then deploy that Windows 7 image to other virtual systems.
If all that you did was click Next through the installation wizard or you did not customize your unattend.xml at all – you actually have a VM that has been installed with two volumes – a System Reserved volume and the volume where the actual system is installed.
The standard process is that you boot your system into the WinPE environment and you use ImageX /capture to create a WIM from a particular volume. Note the word “volume.” ImageX is a volume based tool, not a disk based tool.
The problem comes when we try to deploy this image. We boot into ImageX and /apply – then we reboot the virtual machine and we end up with the error: “BOOTMGR is missing”

The boot manager resides on the System Reserved volume, that is the extra partition that is created but was not captured by ImageX.
According to Microsoft TechNet documentation How to Perform Common Deployment Tasks with Virtual Hard Disks we need to use BCDBoot.exe to “configure the boot entry in the BCD store to be on the volume inside the VHD.” If you follow the link, the portion that we are concerned with is: “Prepare a VHD image to boot inside a virtual machine.”
To repair the VM where the image we need some type of recovery environment. We can get to a prompt to fix the applied image by any of the following ways: Boot to a WinPE 3.0 ISO, Boot to a Recovery Console using the OS installation media, attach the virtual disk to another known working Windows virtual machine.
For my example I have used the Windows Automated Installation Kit and created a WinPE 3 ISO. However, in testing I first simply attached the failing virtual disk to a working VM and performed the same commands. I also found references that booting into the Recovery Console of the installation media also works, but I did not test this. It is important to note that if you use WinPE the bit-ness of your WinPE image must match the bit-ness of the OS in the VM (32-bit to 32-bit / 64-bit to 64-bit).
The repair process:
1. Boot the virtual machine into WinPE (or attach the virtual disk to a running VM).

(An alternative to this is to boot using the installation media.  At the "Install Windows" screen (the first screen of the installer) type Shift+F10 to open a command prompt)

2. Discover the drive letter assigned to the virtual disk that is failing to boot.

3. Execute the following command (pay attention to use the correct drive letter – in my example “C:” is the letter that was assigned to the virtual disk, the WinPE system volume is “X:”).
C:\windows\system32\bcdboot C:\windows /s C:

4. Simply type “exit” to cause WinPE to shutdown and reboot into the VM. If the virtual disk was mounted in another VM, then power off the VM and detach the virtual disk before booting the imported VM.

Just because you can - should you?

This is a question that all administrators must ask themselves at any point in time.

I have know quite a few very creative IT folks in my time, and we can all come up with very clever ideas, combinations, and adaptations of technologies.

Part of my role is to question why.  It is actually part of my job.

I frequently come across things I read by folks and I just think to myself, why the heck would you do _that_?  Just because you can? 

When I worked as an administrator I quickly learned the user dictum:  “Because they can, they will”

Yes, this is generally said in a demeaning way, referring to users, when administrators talk to each other, to their managers, or to folks that write software.

I still say this over and over to the developers I work with.  Generally framed with a statement like:  “If you don’t want them to do that..” or “Of course I entered 300 characters into that field” or “there was no error checking to stop me”.

Think about that as you apply technology or attempt to break technology.  It is all about the intent.

Are you intending to break it?  Do you just not know any better / or not understand it?

If you don’t understand it, then read and ask questions.

As a person who tests software – I absolutely say, yes do it.  But do it in a controlled and smart way.  Pay attention to the entire environment, not just the buttons you are clicking on.

It is usually the greater environment where the real bug exists.

Remote Desktop Simulation Tools

Microsoft has recently released a very interesting tool, that can be useful in scoping the performance and capacity of a Remote Desktop deployment.

It is all Server 2008 and above based – Hyper-V of course and I am sure it is RDS centric, however I wonder how creative I can be in applying its capabilities to other scenarios or hosting systems.  As I can see this being very useful in making comparisons between supporting layers.

The Remote Desktop Simulation Tools

http://www.microsoft.com/downloads/details.aspx?FamilyID=c3f5f040-ab7b-4ec6-9ed3-1698105510ad&displaylang=en

 

A quote from the download page:

The Remote Desktop Load Simulation toolset is used for server capacity planning and performance/scalability analysis.
In a server-based computing environment, all application execution and data processing occur on the server. Therefore it is extremely interesting to test the scalability and capacity of servers to determine how many client sessions a server can typically support under a variety of different scenarios. One of the most reliable ways to find out the number or users a server can support for a particular scenario is to log on a large number of users on the server simultaneously. The Remote Desktop Load Simulation tools provide the functionality which makes it possible to generate the required user load on the server.

A minimal test environment requires:

1. Target Remote Desktop Server
2. Client Workstations
3. Test Controller Host

Terminal Server on Hyper-V

Microsoft is finally (publically) talking about Terminal Server on Hyper-V.

Please note that it isn’t Terminal Server anymore, but “Remote Desktop Services”. This is closer to the way that Citrix talks about desktops and delivery of desktops and applications.

However, in this case it is rather confusing as Remote Desktop Service (without the s) is what has been referred to as the server side of RDP (the protocol) or the remote desktop client.  Just remember, MSFT uses RDP all over the place now.  If it is remote, RDP is involved.

Enough of that, here is the RDS Team blog post I am referring to:

http://blogs.msdn.com/rds/archive/2009/06/24/running-ws08-terminal-server-as-a-virtualized-guest-under-windows-server-2008-r2-hyper-v.aspx

And a snippet:

Running WS08 Terminal Server as a virtualized guest under Windows Server 2008 R2 Hyper-V

One question that the RDS team is asked is whether running Terminal Server virtualized is supported and recommended.  To answer this question we’ve recently conducted some performance testing of this configuration using WS08 Terminal Server running as a guest on Windows Server 2008 R2 Hyper-V.   To answer the first part: this is a supported scenario.

Do you know where your VM has been?

For a tester, creating a situation that causes a piece of software to fail gives you a warm fuzzy feeling inside. Some situations are easy. Some are difficult. Others are accidental, but totally realistic as to how they happened.

I have recently been busy testing virtual disk conversion tools. You might say: "how relatively boring is that.." And I will admit that it isn't the most glamorous thing to test. However, it has lead me to the reality that virtualization of a machine workload is NOT new.

Almost 6 years ago I first began working seriously with virtualization, moving a number of physical machines and their applications into virtual machines on ESX server. Back then, this was really cool stuff. Very few companies were doing it. Microsoft did not officially support running their operating systems within virtual machines, and you could never tell an application support person that you had virtualized the server their application was running on as they would immediately declare "there is your problem."

Well, we have grown and MSFT has gotten into the hypervisor market and all of a sudden this stuff is becoming main stream.

The most important part that I want to bring up is: People are migrating between hypervisor platforms, virtual disk formats, throwing away an old platform for a different one. Why? Because they can.

This is totally relevant to testing, evaluating, understanding performance, and a number of other things with your VMs. What was the life of that virtual machine? How did it get to where it is today? Does anyone in the enterprise even know?

I recently had the pleasure of breaking a piece of software and was told by a developer that "that was a really good bug, you found a great case." Now, as a tester, there is nothing better than a developer giving you a compliment because you found a good bug.

Now, I need to get back to my point.

I had access to a VM that was built on VMware workstation, moved to ESX server, to Virtual Server, to Hyper-V, and then I converted it back to a VMDK from VHD using WinImage.
This is at least five disk format modifications, three tools modifications, and this VM was running an application.

And, during most of those conversions a 3rd party, offline, conversion tool was probably involved. Some examples you may be familiar with: WinImage, Acronis, VMDK2VHD, PlateSpin, etc.

Needless to say there was a problem introduced to the virtual disk at some point in its life. Did any of the tools report this? No. Did the VM boot and run? Most definitely. Did we have any idea that there might be a problem? Definitely not.

It just goes to show you that in this day and age we need to keep track of these things. As who knows where an error can be introduced into an environment.

It is no different than the case of converting that old NT domain controller in the back corner into a VM because there is one legacy application that runs on it that is critical and the application has never been updated... And then you realize that you have migrated between platforms, formats, and who knows what tools.

So, always be mindful of your virtual machines. Their life and lifetime. Not just from a security standpoint, but from a stability standpoint.