Contemplating performant acceptance testing

The symptom

Rack app acceptance tests are slow.

The problem

Acceptance tests add value by exercising the complete system. In doing so, scenarios within acceptance tests reiterate upon previously tested features of the system.

Example

This is, perhaps, the most simple example.

Given an acceptance test that demonstrates the ability for a user to create an account When I write another feature for user, with an account, to authenticate Then my new feature’s acceptance test will require the user to first create an account

Explanation

Most current acceptance testing practices would have us retread this ground. This is the essence of why acceptance testing is slow: most Rack app acceptance tests exercise the entire Rack app stack several times. This is done in order to acrete the desired “setup” state within the test to actually perform the test.

Hypothesis

Why don’t we treat a test suite like a real build process?

  1. Express ordered lists of dependencies between acceptance tests
  2. Each successful test records the changes that it made to the database and the HTTP session
  3. Before executing a given acceptance test, the recorded state changes are played back into the database and http session
  4. Now we can acceptance test as normal

This should allow for acceptance tests to execute a minimal number of trips through the Rack stack. This would, I hope, increase acceptance test execution time geometrically if not by an order of magnitude on the whole.

Your Thoughts?

These are just words: a brain dump of an idea that I had in the shower this morning. I’d love to see (maybe, OMG, write?) a Rack=specific way to facilitate this that could be used by Coulda, Cucumber, etc.

Posted by evan on Friday, August 12, 2011

blog comments powered by Disqus