Tuesday, June 5, 2012

Synchronizing PowerShell script execution on multiple machines

I originally thought to call this “synchronizing watches” but since I am not changing system time I thought it might be misleading.  This post is all about synchronizing script execution on two different machines.

Some folks would use a central client and workflows (PowerShell v3) or PowerShell remoting to execute a script on remote machines at the same time.  Possibly even executing each one as a job so they happen at the same time. 

It also allows me to launch the script on server A and then move to the other side of the lab to launch its script and then begin the repro (or switch the monitor on the KVM).  I know that as long as they share a time service I can be confident in when the tracing began on the remote machine.

In my case I want to execute the script on each machine at the console.  My reasoning is that; the machines are not in the same domain making security messy, or I don’t want to modify firewall rules, or I am capturing network traffic and just don’t want the extra noise to filter through.

Here is my scenario: 

  • I have multiple servers (a large, distributed application)
  • Different things are happening on each server
  • I want to enable a network trace at the same time (to have nicely correlated logs)
  • The systems already have a common time source (the OS handles that after all)
  • I need to manually trigger the reproduction of the bug after the tracing starts, or I need to trigger some event prior to the tracing beginning
  • I only have two hands and two feet

Here is the synchronization snip at the beginning of my script:

Do {
    Start-Sleep ( 60 - (Get-Date -Format ss) )
    
    [string]$nowMinute = Get-Date -Format mm
} until ( $nowMinute[($nowMinute.Length - 1)] -eq "0" -or $nowMinute[($nowMinute.Length - 1)] -eq "5" )

The first thing that I do I describe as squaring up.  I invoke a Start-Sleep but I want to sleep until the end of the current minute, no longer.

Start-Sleep ( 60 - (Get-Date -Format ss) )

You could do this as two lines as well (or three if you like):

$secToZero = 60 - (Get-Date -Format ss)

Start-Sleep $secToZero

Then I get the current minute and evaluate it to the nearest 5 minute block.

I chose the 5 minute mark because it is easy to evaluate to (if the number ends in 0 or 5 it divides by 5).  I cast the returned integer to a string as well, so I can get that last digit.

[string]$nowMinute = Get-Date -Format mm

*Notice that if you use “MM” instead of “mm” you get the Month.

I do my math evaluation at the end of the Until.  Notice the –or between the 0 and 5 after the –eq.  That is important to force the evaluation against both numbers.

Here is where personal preference sets in.  I dreamed up three different ways to handle the evaluation in the Until block.

The first is a very literal evaluation of the second digit in the Seconds to see if it equals “0” or “5”.  the entire last line looks like this:

} until ( $nowMinute[($nowMinute.Length - 1)] -eq "0" -or $nowMinute[($nowMinute.Length - 1)] -eq "5" )

It is the ($nowMinute.Length –1) that puts us at position 1 ( position begins counting at 0) to perform the evaluation.  The position indicator for the array is the square brackets [].

Another way to handle this is through the PowerShell way to handle a Regular Expression (or a pattern match – a nice article on that is here.)

Based on that article I can keep my –or and simply indicate the second digit this way:

} until ( $nowMinute -match ".0" -or $nowMinute -match ".5" )

Or, I can do it more like a regular expression and state in a shorter line that the second digit can match either 0 or 5.  That looks like this:

} until ( $nowMinute -match ".[0,5]" )

The end result is all the same.  Much of the technique is all about personal preference.

3 comments:

Josh Erickson said...

Since you can know the current minute, why not sleep until the next 5 minute cycle?

While a little more complicated, it doesn't need to constantly check what the time is.

With a little finessing, you could change from every 5 minutes to every 10. You could even remote the future datetime calculations and just define an arbitrary datetime object for when you want to wake up.

$now = Get-Date

if([math]::ceiling(($now.Minute/5))*5 -ne $now.Minute) {
$targetMinute = [math]::ceiling(($now.Minute/5))*5;
} else {
$targetMinute = [math]::ceiling((($now.Minute+1)/5))*5;
}

if($targetMinute -gt 59) { $targetMinute = 0; $tmpTime = $now.AddHours(1); } else { $tmpTime = $now; }

$endtime = Get-Date -Year $tmpTime.Year -Month $tmpTime.Month -Day $tmpTime.Day -Hour $tmpTime.Hour -Minute $targetMinute -Second 0
$sleepTime = $endtime - $now

BrianEh said...

Excellent contribution Josh!

Can you elaborate a bit on using [math] and why you are doing that?
(a bit of description around the details of your calculations)

Josh Erickson said...

Glad you like it! And here is the same thing with some comments as requested.


#Store a timestamp so we aren't floating around for all the following operations.
$now = Get-Date

<#
Calculate the next 5 minute interval. ex ceiling(14/5)5 = 15
Dividing the current minute by 5 gives a decimal, (fraction?)...I wish I had
remembered more from math class. Anyway, in C#, it's a float/double/decimal.
So now we have 2.8 (14/5). Using ceiling gives us 3 which we now multiply by
our interval 5 to get 15 minutes.
#>

#If our calculation equals the current minute, we have a problem. That is,
#the time we look forward to is the same time it is now.
if([math]::ceiling(($now.Minute/5))*5 -ne $now.Minute) {
$targetMinute = [math]::ceiling(($now.Minute/5))*5;
} else {
#adding a minute forces us to look forward.
$targetMinute = [math]::ceiling((($now.Minute+1)/5))*5;
}

#Get-Date hates 60 minutes. Reset to 0 and add an hour to our tmp end time.
#Since we only care about everything except minutes, we're dumping $now into a new
#variable that lets us modify the hours if we need to.
if($targetMinute -gt 59) { $targetMinute = 0; $tmpTime = $now.AddHours(1); } else { $tmpTime = $now; }

#Calcuate the next target interval.
$endtime = Get-Date -Year $tmpTime.Year -Month $tmpTime.Month -Day $tmpTime.Day -Hour $tmpTime.Hour -Minute $targetMinute -Second 0

#The ammount of time to sleep til we hit the next interval. Will be slightly off since we just
#did all the above. But it should only be a few milliseconds.
$sleepTime = $endtime - $now