Friday, May 30, 2014

One Language Puritanism for Acceptance Testing ?

A good Web acceptance test - or to be more precise, a tool chain that well-supports end-to-end tests, is apparently quite difficult to achieve.

The problem is not in the high-level languages. Domain Specific Languages like Gherkinm, and hosted language syntaxes, have been used productively.

The main problem is not in the deployment techniques, although lack of closure over the environment is the first major stumbling block to doing any kind of meaningful testing.  Whether it is the local isolated deployment through Ruby On Rails' bundler, rake, and tools like RVM; or CI tool chains like Travis CI, Jenkins, or Bamboo; or just a skunkworks set of scripts, config files, and GIT practices; there are many paths to creating Closure Over a Deterministic Environment  (CODE).

A slightly more interesting problem is generating meaningful and valuable configurations of test data. I've worked with Fixtures and Factories - both are fragile and both take a considerable fraction of the development time to work with consistently. They don't scale all that well, but there is an effective process that can be started and applied in substantive ways across large swaths of functionality.

An even more interesting problem is getting rigorous (meaning: formally translatable) specifications of the expected data and objects. The difficulty is that this takes a lot of long, deep thinking and reflective writing, tactics that were prized activities in decades past but which have met with extreme disfavor in Agile Internet time. Present first-world Homo Sapiens seem to have trained their working memories so that they are barely able to consider meaning in a full length Tweet, let alone a missive as long as this rant.

But I digress. Formal and complete specifications are only plausible, if at all, as an end result of a long development process.  We could expect to possibly accrue such comprehensive specifications over the duration of a project, as an executable or translatable artifact: code; test specification; or literate programming via configuration parameters represented in markup documents. The point is, specifications don't just pop into existence, they are a side-effect of long term thinking and communication processes.

Whether we choose to value that content higher than working project code (which is likely a waste unless it is a research project); or devalue that content to the point of immediately discarding it like so many candy wrappers and beer bottles on the ground (which is a likely waste unless it is a trivial project); or find a way to make that content an integral aspect of the artifacts under construction (literate programming, configuration by convention, specification translation); so long as the team can maintain an adequate level of context and understanding, there is a path forward.

No, the devil in acceptance testing is in the details: our tools on the client side are immature, and one reason that people are stuck on stupid is an irrational obsession with using one, and only one, language to solve the problem.

Don't get me wrong: that language you use, be it Javacript or PHP or Gherkin or whatever, is just fine. But that language choice, for end-to-end acceptance testing, is barely even relevant.  The specific language choice is not necessary and not sufficient.  What matters is whether the tooling can actually perform a simple step like input a select option and observe a change happen as a result of a back-end transaction. I recently ran into trouble with Codeception/Behat/Mink because of this issue. Most of the acceptance testing frameworks I've seen give the developer way too much grief to work around these scenarios, if they can even work with them correctly at all.

I'd rather use Javascript (or better yet, CoffeeScript) to write acceptance tests for a PHP application, if it means that the test engine will be able to smoothly traverse the client side interactions. Codeception is a nice integration of PHP toolkits - one of the best I've seen - but the fact that a test framework uses PHP exclusively as a testing language means LESS THAN NOTHING to me. That a client-side test engine which understands Javascript can be bridged to a library of modules in a second language (PHP) simply presents on-going risk of flaws with no opportunity for accruing benefits.

It really amounts to introducing multiple layers of indirection in order to avoid using a native language of the actual test platform. As always the problem this strategy creates is too many layers of indirection.