Refactoring Proposal: Plugins And DEVELOP

There's an issue with the current DEVELOP codebase and the plugins. Several plugins use functions and variables not published through the Func API. This problem is endemic; even the InterWikiPlugin abuses the interface.

This is not really the fault of plugins authors. It is because the Func interface has never kept pace with what plugins need to do, except in a few isolated cases where the plugin author was also a core developer. Note that this shows how inadequate the whole "Func interface" approach has been. It is not (can never be) respected by plugin authors as the only interface.

There are some cases where functions used by plugins no longer exist in the core, because they have been refactored to new homes. This raises the question of how to handle the fact that some key plugins will no longer work with the DEVELOP codebase.

As I see it there are three of possible approaches:

1. Recode plugins to switch on an increments Plugins::VERSION number and call the new (still illegal!) APIs
2. Add a DakarContrib compatibility module, similar to the CairoContrib
3. Code stubs to the moved functions into the core.

1 requires changes to most plugins. Since many plugins are owned by sleeping contributors, this is probably not an option. 2 would work, but it is less than ideal. 3 is the simplest approach. However it has a major problem. It means that the core will be carrying "dead" code forever, unless a decision is taken to rationalise the plugins.

Many plugins are dead, but we currently have no way of knowing which ones. Even plugins shipped in the release are not used in many installations; for example I regularly have to uninstall smilies, interwiki, table, edittable and spreadsheet plugins in sites where performance is more important than those plugins.

My proposal for the way ahead (for Lynnwood's roadmap, if you like) is to:

1. Use the PluginAppraisals to establish which plugins are used. An unappraised plugin can be assumed to be dead, after a reasonable period.
2. Continue refactoring TWiki modules to an OO interface. These OO interfaces get published to Plugins.
3. At some future release, dump Func, recode the plugins, the ones that are actually used, to the new interface, and ditch the rest.

-- CrawfordCurrie - 13 Nov 2004

"Tuez-les tous; Dieu reconnaitra les siens."
("Kill them all; for the Lord knoweth them that are His." Popularly translated as "Kill them all; let God sort them out")

[Arnaud-Amaury, Abbot of Citeaux, 1209, when asked by the Crusaders what to do with the citizens of Beziers who were a mixture of Catholics and Cathars.

See also http://c2.com/cgi/wiki?CureWorseThanTheDisease for Yourdon's observations on this to s/w development (complete with obligatory reference to Dilbert).

Why is this observation relevant?

Two points:

1. The Cathars were know as "The Perfect Heretics". Seeking a "perfect" solution, not least of since software is far from being an engineering science is .... not going to lead to an easy consensus. Lets hope it doens't lead to a fork.
2. "The ones that are actually used" will be recognised by that fact - the TWiki God's will notice. The rest get dumped - aka "killed".

-- AntonAylward - 14 Nov 2004

Let them break. Guarantee that shipped plugins work, and let the other plugins that perform "illegal" operation to be repaired if someone need it.

At least that's my first impulse. But that has the potential to prevent people to upgrade to the latest version because some unpublished homemade critical plugin will stop working, so a compatibility module seems to be the most sensible thing to do.

and, from this point onward, PLEASE PLEASE PLEASE let's define a good, complete, robust and stable Plugin API.

-- RafaelAlvarez - 15 Nov 2004

IMHO we can never define a "good, complete, robust and stable" API. The reason is simple; we never know what the next guy will want to do. We already have multiple authentication implementations and multiple Store implementations. Plugins already call some really quite obscure functions in the core, often for quite obscure reasons. It is quite simply impossible to define an "API" that provides everything anyone will ever need.

A truly pluggable architecture lets you exchange any component of the system without breaking any other piece. We are a long way from that, but it should be our goal.

So, what is the future for the current Func API? Well, it's my belief that it is useful, and will be useful for some time to come, as a buffer between plugins and the core. However, I think the rules need to change. First, the Func interface needs to expand as and when required to do so by plugins. We should not be afraid to add new functions. If the functions are there, we can always re-implement them if we change the core; if they are not, plugins will call past them and we're stuffed.

Note that we have already proven this approach using Marzia's ModPerlize approach. it is possible to ModPerlize the core and still retain fully functional existing plugins despite a radical change in the core.

-- CrawfordCurrie - 15 Nov 2004

if you want to retain Func, why not move it to TWikiPlugins, so when a plugin author needs access to something, they can simply add support. this won't work for adding handlers, but is it a start? Mind you whenever i needed / wanted to write a plugin, i'd just make it work, and Func was always furthest from my mind.

-- SvenDowideit - 15 Nov 2004

Crawford: I agree that we should not fear to add new functionality to Func. My primary concern, and the reason for my pledge, is that I feel that plugins (both in Twiki.org and in the Wild) are one of the chains that are preventing Twiki to move forward (see the arguments of broken application agains L^3).

Sven proposal seems a sensible solution, sharing the FuncDotPm module maintenance with the PluginsCoreTeam. Again, for backward compatibility with older plugins the FuncDotPm module should remain in place (if only delegating to the Plugins::Func module).

-- RafaelAlvarez - 15 Nov 2004

this is an opportunity to get all of the plugins into cvs and force them through the automated plugins build and upload. once in CVS, i imagine we'll need to use the same BranchTagNames as in SVN. additionally, each plugin could be checked tagged on an AthensRelease, BeijingRelease, or CairoRelease branch (along with MAIN or DEVELOP?). in short, i feel that plugins development is going to have to adopt the same branch development strategy as the core.

-- WillNorris - 15 Nov 2004

Whither Func?

Lately I'v been working on OO-ifying TWiki (see TWikiOO) by creating singleton objects for each of the code modules. This means that no code can be called without a pointer to an object, either the TWiki "session" object or one of the objects for the other modules. Now, the Plugins interface is able to keep working the way it always has done because each of the plugin handlers sets a "local" global variable (one localised to the current call stack) before calling any handlers. This means that any call-back from the plugin to Func methods will be able to pick up the session object, and so access the functionality of the rest of TWiki.

All this has highlight to me that the Func interface model - while worthy for the working model when it was introduced - is no longer a good interface. There is no reason to remove it, but there are good reasons not to build new code that uses it.

1. Func does not expose some key methods that plugins need to safely access the store.
2. Plugins using Func will work with mod_perl, but it is risky - if a plugin has any global variables, they may end up being shared between different invocations of the plugin. It would be much better to push plugins into the OO model.
3. The procedural nature of the Func interface makes it difficult and dangerous to extend.
4. Func cannot be called outside stack below a plugin handler. This means that code using Func can only be used in that context.

The proposal is therefore to:

1. Identify an OO Plugins TWiki interface, based on the methods published from the TWiki object and the other system objects (Store, Form, Search etc). This will expose a lot more of the functionality of the core, without exposing implementation.
2. Deprecate (though keep) the Func interface.

I believe this is a major step forward, and should be done for the Dakar release.

-- CrawfordCurrie - 12 Jan 2005

Nice idea: One comment: I think it is better to tag (by a comment) functions/methods in the code that people can use rather than making a Func layer that people will forget to maintain and will lag behind in functionality. Just like documenting functions in the code will be kept more up to date than documentation in a separate file.

-- ColasNahaboo - 13 Jan 2005

That's exactly what I intended, Colas. It really should be done in conjunction with a polished "OO TWiki Plugin" implementation, but I haven't had time to think much about that yet.

-- CrawfordCurrie - 13 Jan 2005

This is ongoing work, so I unassigned it from Dakar until there are specific proposals.

-- CrawfordCurrie - 16 Feb 2005

Conclusion for Dakar was to fix the ones that people are using, and let the rest break.

Proposal:

• Create a TWiki::Api package, that serves as a "stable" layer above the rest of the TWikiCore. This "layer" can also provide some common services, like logging and audit-trailing.
• Publish commonly used objects (Meta and User come to mind) via this layer.
• Make all Plugins and TWikiCore modules to reference this API package instead of directly calling other modules.
• Deprecate Func.pm, first by delegating all the calls to the API layer and at some point in the future removing it.

-- RafaelAlvarez - 13 Feb 2006