Friday, May 25, 2012


XML test case definitions

If your test framework reads XML files to load a list of suites and test cases, it's defective.

XML is not a human writable format. It's for systems to communicate with other systems. Why would you select XML for a file that's supposed to be created by humans?

I take issue with the entire premise of having a configuration file in the first place. The rationale usually goes something like this:
  1. We'll create test cases that have parameters that make them flexible
  2. An XML file will allow us to call those test cases in new and unusual combinations
  3. We'll use different XML files for different types of test runs
This should all be in code. Suppose I have a test case like this:
def test_creating_directories(depth):    # Some test code
I might have an XML file containing:
But why? This XML file is going to get loaded up and added to data structures at test time. Why not just create those data structures in the test language? What's the point of using XML here at all? Create these data structures in your code directly:
tests = [
    (test_creating_directories, 3),
    (test_creating_directories, 4),
    (test_creating_directories, 5),
My point here is not just that this is easier to read and write. Once you start seeing this as a data structure, you realize you can do something like this:
tests = [] for i in xrange(3,6):     tests.append((test_creating_directories, i))
Without doing something seriously hacky, you'll never have loops and other programatic test case definition in an XML file.

The only slight exception to this rule is for compiled test cases - so this would apply to folks writing in C, C++, C#, Java, etc. In those cases you might want to change the list of tests near runtime rather than at compile time. But for a language like Python, where you can just edit the files, there's no reason to use these external test suite data files.

Tuesday, May 08, 2012


Test types

Although various organizations have tried to create standard test types, it's still the case that when you start with a new QA team, you need to understand what functional, smoke, sanity, and acceptance tests mean. It seems that there are three schools of thought for categorizing a test case:

  1. Time based - This is sometimes defined as how long each test takes to run, or how long all tests of a given type take to run. A post on the excellent Google Testing Blog by Simon Stewart says that Google uses the amount of time a test takes to run, along with complexity metrics like threading and network access to categorize each test. The full article is here.
  2. Functionality/coverage based - A lot of organizations will use tiers or buckets to define tests. An example might be that sanity is the bare minimum functionality the product needs to exhibit to be considered worthy of further testing. Smoke is exploring the common options/use cases customers will use "by default". Functional is everything else. This article at differentiates sanity and smoke in terms of what the test cases cover.
  3. Triage based - Writing and executing test cases is great, but we have to factor in the cost of analyzing the results. Even if you have sophisticated links to your bug tracking database from automation, test execution creates work. Organizations often set targets like "sanity test failures must be analyzed within two hours" or "we work through functional test results at least once a week." In this way, the triage behavior defines the test type.
These schools of thought can be applied in tandem, sometimes including all three. One group I worked in at VMware borrowed attributes from all three areas in defining Build Acceptance Tests: tests which ran in under two hours, covered all major use cases of the product and were triaged daily.

It's easy enough to find quibbling over time and functionality based definitions for test types, but much harder to find talk about triage. My opinion is that the industry has not done a great job of thinking about the business side of software testing. We automate everything and omit the ROI. Triage costs are the biggest variable cost in software testing yet the groups I've been a part of spend very little time thinking about it. I'd love to hear from folks with different experiences here.

Sunday, June 05, 2011


Version 2.0.0 of Wordpress get-post plugin

Well, after a couple of years I dusted it off and rewrote a lot of the guts. It's about a hundred times shinier. Download at

Source at

Friday, May 13, 2011


How to test a proxy

echo test | nc -X connect -x proxy_host:port target_host target_port

Tuesday, May 10, 2011


Belated announcement: First public release of PyATOM

The PyATOM team is proud to announce the initial release of PyATOM.

About PyATOM:

Short for Automated Testing on Mac, PyATOM is the first Python library to fully enable GUI testing of Macintosh applications via the Apple Accessibility API. This library was created out of desperation. Existing tools such as using appscript to send messages to accessibility objects are painful to write and slow to use. PyATOM has direct access to the API. It's fast and easy to use to write tests.

Changes in this release:

It's the first public release, so none yet!

Special thanks:
The VMware Fusion automation team
Nagappan Alagappan and the LDTP team

Download source:

Documentation references:

Documentation is still a work in progress. Read the README on the
Github page for an introduction to PyATOM.

Report bugs -

To subscribe to PyATOM mailing lists, visit

IRC Channel - #pyatom on

Friday, January 28, 2011


Python exception madness - err, Python namespace madness

Riddle me this: What the heck?
import b

class MyException(Exception):

if __name__ == '__main__':
   except MyException:
      print 'Caught my exception'
   except Exception as e:
      print 'Caught something else: ' + repr(type(e))
from a import MyException

def evil():
   print 'Raising my exception'
   raise MyException('My exception')
$ python
Raising my exception
Caught something else:
<class 'a.myexception'>

What?! Looks like an oddity of the Python namespace. Note the class being listed as 'a.myexception' - it's as though the imported exception class isn't being recognized as being equal to the class defined just a few lines above. If you put the exception class in, it works.

I'm guessing this is documented behavior of the package import mechanism... but it was certainly unexpected behavior to me.


My good friend Rick suggested this was because of circular imports, but I suspected it was because a was both __main__ and a. So I did this:
import a

import b

class MyException(Exception):

def test():
   except MyException:
      print 'Caught my exception'
   except Exception as e:
      print 'Caught something else: ' + repr(type(e))
import a

def evil():
   print 'Raising my exception'
   raise a.MyException('My exception')
$ python 
Raising my exception
Caught my exception
To distill a lesson from all this, it's: don't put exception classes in any Python scripts run directly... Or maybe don't import a Python script that's run directly from another Python script.

Rereading all this, I think in the first case that a might actually have been read in twice - once as __name__=='__main__' and once as __name__=='a'. In that case, it seems that __main__.MyException is not a.MyException - which leads to the exception handler failing.

If this is one of the reasons people recommend against circular imports, I'm inclined to agree - I spent way too long troubleshooting a very vague issue that seemed like an interpreter bug at first.

Thursday, October 14, 2010


James' guide to sshing through an http proxy

May you never need this guide.

Preparing sshd on the server:

Why port 443 (https)? https can't be truly proxied without subterfuge. Between the SSL protocol itself and the presumed good behavior of the CAs, your company or school proxy can't establish trusted https connections on your behalf. To overcome this, http proxies implement a command called CONNECT which establishes a pass-through connection between the client and the specified host. The operating theory is that the vast majority of http proxies are going to allow CONNECT to arbitrary hosts over port 443, or the web would be broken for most of their users. The proxy can't interfere with this traffic and hopefully they don't look at it too closely either, because ssh is distinguishable from ssl traffic.

Client side (for testing):
  • There is a program out there called corkscrew but it isn't needed on an OS with a relatively thick GNU stack, like Linux or OS X. It needs to have nc, which is corkscrew on crack.
  • Try this: ssh -o ProxyCommand="nc -X connect -x <proxy host>:<proxy port> %h %p" -p 443 <user>@<ssh host>
If that works, awesome! Toss the ProxyCommand line in ~/.ssh/config. The Internets have some resources on doing this dynamically based on whether you're behind a proxy or not.

This page is powered by Blogger. Isn't yours?