Tuesday, May 08, 2012

 

Test types

Although various organizations have tried to create standard test types, it's still the case that when you start with a new QA team, you need to understand what functional, smoke, sanity, and acceptance tests mean. It seems that there are three schools of thought for categorizing a test case:

  1. Time based - This is sometimes defined as how long each test takes to run, or how long all tests of a given type take to run. A post on the excellent Google Testing Blog by Simon Stewart says that Google uses the amount of time a test takes to run, along with complexity metrics like threading and network access to categorize each test. The full article is here.
  2. Functionality/coverage based - A lot of organizations will use tiers or buckets to define tests. An example might be that sanity is the bare minimum functionality the product needs to exhibit to be considered worthy of further testing. Smoke is exploring the common options/use cases customers will use "by default". Functional is everything else. This article at softwaretestinghelp.com differentiates sanity and smoke in terms of what the test cases cover.
  3. Triage based - Writing and executing test cases is great, but we have to factor in the cost of analyzing the results. Even if you have sophisticated links to your bug tracking database from automation, test execution creates work. Organizations often set targets like "sanity test failures must be analyzed within two hours" or "we work through functional test results at least once a week." In this way, the triage behavior defines the test type.
These schools of thought can be applied in tandem, sometimes including all three. One group I worked in at VMware borrowed attributes from all three areas in defining Build Acceptance Tests: tests which ran in under two hours, covered all major use cases of the product and were triaged daily.

It's easy enough to find quibbling over time and functionality based definitions for test types, but much harder to find talk about triage. My opinion is that the industry has not done a great job of thinking about the business side of software testing. We automate everything and omit the ROI. Triage costs are the biggest variable cost in software testing yet the groups I've been a part of spend very little time thinking about it. I'd love to hear from folks with different experiences here.

Comments:
I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.
Hadoop Training in Chennai
 

Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us.


Hadoop Training in Chennai
 


That is very interesting; you are a very skilled blogger. I have shared your website in my social networks..!


Online Reputation Management
 


Thanks admin for sharing the unique content, you have done a great job

Installment loans in Mississippi
Payday loans in Mississippi
Title loans in Mississippi
 

Wonderful blog.. Thanks for sharing informative Post. Its very useful to me.

Installment loans
Payday loans
Title loans
 
Post a Comment

Links to this post:

Create a Link



<< Home

This page is powered by Blogger. Isn't yours?