Friday, November 26, 2010

Industrial Strength Testing for Joomla Part 3

Under a test-first development practice when you're rolling out an add-on, you should also be rolling out a rigorous suite of tests to demonstrate that the add-on really works and satisfies some fundamental security and accessibility criteria.

Lacking such an automated test suite, it is not rational to expect an add-on to perform correctly. It is certainly not reasonable to purchase an add-on without the assurance that a test suite passes and can be repeated under your host environment. 

The truth is, most software is crappy. Let's posit a valuation function Crppnss(t) to determine how crappy a piece of software is, and suppose it is inversely proportional to the number of automated tests V(t): Crppnss(t)=1/V(t). That's a reasonable assumption. Although there are studies that suggest the relationship between latent bugs and number of tests is linear, even a small number of tests can have an impact much greater than a linear correspondence.

Since most Joomla components show little or no evidence of automated testing, it follows that most Joomla components should have high Crppnss. That is to say, the theory would predict that they should be quite crappy, and that is just what we see in practice.

Am I missing something here? No doubt. Yet I suspect that the Joomla/PHP testing cup will still be less than half full when I'm filled in on what I'm missing.

After my initial draft of this post, I did more research. Apparently PHPUnit and Selenium, with a Joomla specific runner, is the soup du jour for Joomla 1.6 testing. It is evident that the team has put in a good amount of effort to put together a rational tool chain and process. So the Crppnss for Joomla has officially gone from infinite in version 1.5 to less than one in version 1.6. 

No comments: