Monday, February 2, 2015

Testing and Checking in the cloud service era

Services.  DevOps. Cloud Services. Rapid prototyping. Start-up methodologies. Agile development.

These are all terms that we hear in the software business.  And they generally all point to what some folks consider a reduction in testing.  And what many customers consider a reduction in overall quality.

If you follow some of the big names in the software testing world ( Michael Bolton, Rex Black, James Bach, and others ) one theme that resonates with all of this is testing vs checking.

And in my experience what I see happening in this cloud service era and the rise of DevOps is that there has been a shift away from 'testing' and a greater focus on 'checking'. 
Some say that this makes the developer more accountable instead of them blaming test for missing it.  I don't argue that quality was always the developers responsibility anyway.

In my words, checking is more like unit testing.  Did this thing respond in the way it was designed to?  Everything is positive, everything is looking for the intended desired outcome.

Where testing is checking plus looking at and driving appropriate failures.  Not only did this thing do the positive action, but when I send a negative action - what happens.  Did it fall over, did it respond with a proper error, did something totally unexpected happen.  And then there are other studies; load, scaling, fuzzing, chaos monkey, etc.

In fact, I would argue that in this cloud service world, this shift to a greater emphasis on checking is actually a bad thing.  I tis good for development in that it gets more code out the door, and in theory more features and quicker fixes.  However, it ends up making for very fragile applications, and I know of few cloud applications that don't have a high number of dependencies on other cloudy services.

Now, why does this matter?  Because, customers have an emotional connection to your product.  Not a factual relationship with it.

Any small issue that blows up into a large issue, or a planned two hour outage that becomes a 4 to 8 hour outage impacts the feelings of the customer in regards to your service.  This impacts their perception of your quality.  As the customer sees quality as a value judgment.  No different than excellent service at a restaurant.  The more expensive the restaurant, the better everything must be.

This gets messy in the cloud service era.  Because of pricing competition.  The model is closer to that of a bank.  If a customer only consumes one of your services, it is easy to switch.  So you upsell them with more and more services and possibly take a loss in doing it.  This generates a type of lock-in as it becomes more and more difficult for a customer to leave and go somewhere else.  The tipping point.

Anyone who has been involved in IT purchasing decisions knows that software is brought into the enterprise through some corporate division.  Then it gets handed to IT to evaluate.  And in very rare cases, IT actually gets to give feedback or get the corporate division to consider alternate options.
The other side to this is that IT is the one doing the evaluation and choice.  When this is the case the judgment is all about the getting started experience - if I have to crack a manual to get started, something is not right.

Again, checking would focus on the binary functions of the software where testing should be looking at the overall experience.
Just something to keep in mind in this rapid software era.

No comments: