Several times I've deservedly received flak for failing to fully test code contributed for patches. Some of that flak is surely deserved; I have no excuse for cavalier coding. But I am also acutely concious that there are code changes that I have made that are still not adequately tested. The reasons for this are, I think, threefold:

First, there are very few tests or specs defined for existing code, and very little documentation, so I can't always deduce what the code is supposed to do. That means that when I write tests, I can't test for the right behaviours. There is no culture of automating tests in the TWiki project, so tests have to be repeated manually, and if you don't know what to test for, that can be very hard. I am concious, for example, of the number of CGI parameters I had to leave undocumented during the refactoring of the UI scripts, simply because I couldn't figure out how they were supposed to be used. The "undocumented cmd parameter" is only the tip of the iceberg here.

Second, I find writing tests for TWiki code exceptionally difficult, because of the widespread use of global variables, side effecting functions, overlaps between modules, and functionality duplicated in several places. There are of course initiatives in place to make this better - Walter's refactorings have helped a lot - but it's still very difficult to write an adequate test for, for example, the Store module - which should be about the simplest module to test in the whole of TWiki.

Third, I find testing perl to be somewhat unpleasant; the number of things that can potentially go wrong in - for example - regular expression matching is huge, and identifying representative corner cases is very difficult. I'm slowly getting better at it, though that may be a result of increasing familiarity with the TWiki code.

Of course, none of this excuses cavalier coding, or poor documentation on my part; better discipline would of course improve things. However if I encounter these sorts of problems, I hate to think what it must be like for a less tenacious programmer coming into the project! This may well be a reason why people are reluctant to contribute.

There are a few things that all contributors can do to improve this situation. First off, when you contribute a patch, or edit some code, take a bit of extra time to document the undocumented code that you had to understand in order to generate the patch. Don't just wait for someone else to do it. Second, if tests are defined for a module, then run the blessed things after you have finished your changes. Third, write new tests wherever they seem appropriate. The great testing initiative proposed after Beijing never seems to have materialised; we have added a lot of new functionality to Cairo without improving the test picture at all, AFAICT. Finally - and this is bound to cause an argument - use automation to ensure development steps can't go wrong. There really should be no manual steps involved in building and testing the production system, so that any patch contributor can have the full source-to-product cycle on their local machine.

-- CrawfordCurrie - 12 May 2004

Crawford, thanks for the mea culpa. But we all know that you are among the most conscientious contributors when it comes to testing the code. Your plugins come with test code which is much more than any other plugin does. (You are not the best documenter, but that is a different story ).

Again, thanks for the great job you do, and for reminding us of the importance of testing.

Just one question: were should one document code that one figured out? Where to look whether something is documented or not? Are you refering to comments in the code?

-- ThomasWeigert - 12 May 2004

The lack of enforced encapsulation around the TWiki core modules makes this far worse that it could be. Plugin authors do their best to work inside the framework but often find FuncDotPm inadequate and resort to making calls outside it. They do this because they can and indeed it is better that they do than them coding the same functionality in their own module.

However, this is a poison pill that constrains the flexibility of the project going forward. You only need to see the mess revealed by the PluginsConformanceReport to realise that the poor encapsulation from day 1 has made enforcing this today impossible.

There are two parts of a strategy that could help here:

Hard encapsulation for testing

This mandates that people writing code turn off the forgiver (below) and turn on strict encapsulation to make sure that any call made to a sub outside of TWiki::Func fails.

Forgiving encapsulation for runtime

This would entail writing an adapter to catch any call to a subroutine in an internal modules where the subroutine does not exist. Instead of throwing an exception it can check to see whether this call used to exist and re-route it in the interim to the correct place. At the same time, we log that it has happened, what the offending caller was and use the log to improve the code.

-- MartinCleaver - 12 May 2004

There are too many things you could not do with the Beijing API. Maybe things have improved, for Cairo but clearly Func.pm was not sufficient, as evidenced by many of the API violations.

Also, Crawford's point, I think, is not limited to plugins.

-- ThomasWeigert - 13 May 2004