Friday, May 25, 2012


XML test case definitions

If your test framework reads XML files to load a list of suites and test cases, it's defective.

XML is not a human writable format. It's for systems to communicate with other systems. Why would you select XML for a file that's supposed to be created by humans?

I take issue with the entire premise of having a configuration file in the first place. The rationale usually goes something like this:
  1. We'll create test cases that have parameters that make them flexible
  2. An XML file will allow us to call those test cases in new and unusual combinations
  3. We'll use different XML files for different types of test runs
This should all be in code. Suppose I have a test case like this:
def test_creating_directories(depth):    # Some test code
I might have an XML file containing:
But why? This XML file is going to get loaded up and added to data structures at test time. Why not just create those data structures in the test language? What's the point of using XML here at all? Create these data structures in your code directly:
tests = [
    (test_creating_directories, 3),
    (test_creating_directories, 4),
    (test_creating_directories, 5),
My point here is not just that this is easier to read and write. Once you start seeing this as a data structure, you realize you can do something like this:
tests = [] for i in xrange(3,6):     tests.append((test_creating_directories, i))
Without doing something seriously hacky, you'll never have loops and other programatic test case definition in an XML file.

The only slight exception to this rule is for compiled test cases - so this would apply to folks writing in C, C++, C#, Java, etc. In those cases you might want to change the list of tests near runtime rather than at compile time. But for a language like Python, where you can just edit the files, there's no reason to use these external test suite data files.

Tuesday, May 08, 2012


Test types

Although various organizations have tried to create standard test types, it's still the case that when you start with a new QA team, you need to understand what functional, smoke, sanity, and acceptance tests mean. It seems that there are three schools of thought for categorizing a test case:

  1. Time based - This is sometimes defined as how long each test takes to run, or how long all tests of a given type take to run. A post on the excellent Google Testing Blog by Simon Stewart says that Google uses the amount of time a test takes to run, along with complexity metrics like threading and network access to categorize each test. The full article is here.
  2. Functionality/coverage based - A lot of organizations will use tiers or buckets to define tests. An example might be that sanity is the bare minimum functionality the product needs to exhibit to be considered worthy of further testing. Smoke is exploring the common options/use cases customers will use "by default". Functional is everything else. This article at differentiates sanity and smoke in terms of what the test cases cover.
  3. Triage based - Writing and executing test cases is great, but we have to factor in the cost of analyzing the results. Even if you have sophisticated links to your bug tracking database from automation, test execution creates work. Organizations often set targets like "sanity test failures must be analyzed within two hours" or "we work through functional test results at least once a week." In this way, the triage behavior defines the test type.
These schools of thought can be applied in tandem, sometimes including all three. One group I worked in at VMware borrowed attributes from all three areas in defining Build Acceptance Tests: tests which ran in under two hours, covered all major use cases of the product and were triaged daily.

It's easy enough to find quibbling over time and functionality based definitions for test types, but much harder to find talk about triage. My opinion is that the industry has not done a great job of thinking about the business side of software testing. We automate everything and omit the ROI. Triage costs are the biggest variable cost in software testing yet the groups I've been a part of spend very little time thinking about it. I'd love to hear from folks with different experiences here.

This page is powered by Blogger. Isn't yours?