Refactoring the Plugin API

This topic covers two aspects of the Plugins API; refactoring it to be "more OO", and refactoring it to make it more applicable to the job in hand.

For an analysis of plugin API use in published plugins, see AnalysisOfPluginsAPIUsage (script generated, so don't bother editing it!)

Refactoring to be "more OO"

I'd like to introduce a new type of plugin, which implements an object-oriented interface rather than using the procedural model the current ones do. PluginsDotPm could detect which was which and act accordingly.

I'd also like to add some hooks to preload ModPerl-aware plugins in that environment, allowing them to be up and ready to go for subsequent requests.

AndreaSterbini discusses in EnhancementsToThePluginAPI idea of having initPlugin return an object rather than a success flag, thus allowing a backwards-compatible method for introducing object-oriented plugins. This is a good idea, and I think I'll use it as the kernel for implementing my changes. Plugins will be strongly encouraged to use the new model, but will not be required to do so.

One thing that all plugin developers should be aware of for ModPerl is persistent globals, and the need to reinitialize them per-request. Especially on Apache 2.0, whose mod_perl instances may be quite long-lived if one of the threaded MPM's is in use.

Refactoring to be more useful

Another way to view this data is as a source of inspiration for missing functionality in TWiki::Func. While some usage indicated in AnalysisOfPluginsAPIUsage is surprising, given what is provided in TWiki::Func, there are needs expressed which should be taken into consideration for extending the API afforded by TWiki::Func. In particular, this seems to be true for

• Handling of forms
• Interfacing to mail
• Saving a topic
• Dealing with topic locking
• Dealing with meta data

Contributors: -- ThomasWeigert -- WalterMundt -- CrawfordCurrie

---

Discussion

Walter, from a plugin author's perspective it's far more important to get rid of the Func interface, ASAP. Plugins need OO interfaces to the parts of TWiki that they can negotiate with. Without them, plugins authors are forced to continue poking around in the guts of TWiki, duplicating code and referencing unpublished functions and variables.

Here's a complete implementation of a simple plugin that works this way.

{ package TWiki::Plugins::MyPlugin;
  @TWiki::Plugins::MyPlugin::ISA = ("TWiki::PluginAdapter");

  # PUBLIC
  # replaces initPlugin
  # Invoked through $pluginObj = new TWiki::Plugins::MyPlugin( $twiki )
  sub new {
    my ( $class, $twiki ) = @_;
    # The $twiki object is used to recover topic, web, user, installWeb
    my $topic = $twiki->{topic}; # pointless, just to illustrate
    my $this = bless( $class->SUPER::new( $twiki ), $class );
    $this->{debug} = $twiki->getPreferences( "MYPLUGIN" )->getValue( "DEBUG" );

    return $this;
  }

  # PUBLIC
  # Override TWiki::PluginAdapter (which in turn implements TWiki::PluginInterface with empty stub functions)
  # Invoked through $pluginObj->commonTagsHandler( $twiki )
  sub commonTagsHandler {
      my ( $this, $twiki ) = @_;

      # Get reference to text
      my $text = $twiki->getText();
      $$text =~ s/%INCLUDE{.*?}%/&_handleTag( $this, $1, $twiki )/geo;
  }

  # PRIVATE do your stuff
  sub _handleTag {
    my ( $this, $params, $twiki ) = @_;
    my $attrs = new TWiki::Attributes( $params );
    my $web = $attrs->get("web") || $twiki->{web};
    my $topic = $attrs->get("topic"} || $twiki->{topic};

    return $twiki->getStore()->getTopic($web, $topic)->getText();
  }
}

1;

-- CrawfordCurrie - 24 Feb 2004

It is stated above that:

• Walter, from a plugin author's perspective it's far more important to get rid of the Func interface, ASAP. Plugins need OO interfaces to the parts of TWiki that they can negotiate with. Without them, plugins authors are forced to continue poking around in the guts of TWiki, duplicating code and referencing unpublished functions and variables.

There's a lot to be said for OO, but how critical is this really? I would suspect that a few more functions added to Func.pm would considerably reduce the need to go straight into the guts of TWiki. Any OO interfaces will fundamentally have the same problem as Func.pm, the more you expose, the more difficult it is to change the underlying code (OO can help in this, but I don't think it makes much differences in this case).

On a historic note, an OO version of Func.pm was considered and rejected. I am a big fan of OO (though not so much of Perl OO, I found doing the Meta part of TWiki in Perl a bit of a slog compared to, say, Java). But, you certainly need a good OO design if it's to fly. This isn't completely obvious for TWiki. Still it would be an easy job to propose and write an initial thin OO interface to TWiki, then demand could indicate if this should move into the core. The example above may have performance issues e.g. the call $twiki->getText()

-- JohnTalintyre - 24 Feb 2004

Critical compared to what? Compared to making plugins themselves objects, it's far more critical. You're going to hate me for this, but I am going to argue that the interface(s) published to plugins are the most critical aspect of the whole TWiki project. Not just because plugins - the major source of contributed code in the TWiki project - depend on it, not just because much of the functionality in the core should be beyond it, but because those interfaces are a focal point, from which OO design and good practice can spread both directions through the rest of the code.

As for "demand" indicating whether it should move into the core; John, I and others have been politely pressing that "demand" in various ways for over 2 years now.... and I'm sorry, performance issues in $twiki->getText()? A pointer dereference? Performance issues compared to what? If you really have a problem with that, then publish $twiki->{text} as the interface instead.

BTW, writing a class that subclasses the plugins interface is exactly how KWiki does plugins.

-- CrawfordCurrie - 24 Feb 2004

Hey Crawford, while I am proud that my plugin comes out as the Bad-est there are at least 3 plugins that contain TWiki.pm with modifications (and thats one). And worse, those modifications are for the Beijing release, and not relevant for the CVS version..

otherwise - damned interesting reading

-- SvenDowideit - 26 Feb 2004

Why was the readTopic function deprecated from the plugins API? It leaves the plugin author with an insurmountable problem; viz, how do they read a topic without embedded meta-data? At the moment, authors have to s/^%META.*$//g or similar - which is hardly portable. Further, readTopicText is bad news because it assumes that META is interleaved with text - which is true with the current Store implementation, but probably false for any other implementation.

I propose that readTopicText is deprecated, readTopic is brought back and readTopicMeta is introduced, that returns a Meta object. The Meta interface is fairly sound, and should be published.

-- CrawfordCurrie - 28 Feb 2004

I'd agree that we need something better. personally i'd like plugin authors to be able to request a modification of a topic without necessarily needing to read topic themselves.

ie. CommentPlugin just calls modifiyTopic( "Codev.RefactorPluginAPI", "%COMMENT%", "I like the idea --SD\n\n%COMMENT%"); where there is a regex that replaces the occurance of the second text with that in the third parameter.

if we can have the same for editing meta data then plugins (and parts of the core) can be totally ignorant of the data's location and format.

what do you guys think

(in the short term, i concur with Crawford.)

-- SvenDowideit - 29 Feb 2004

Arrghhh no Sven, I hate it - far too complicated. KISS. I want/need access to the whole of the topic text, even if I may only be changing a small part. For example, I may need to count the number of previous occurences of a tag in the topic before I reach the one I'm interested in......

-- CrawfordCurrie - 29 Feb 2004

sorry, i didn't mean that this would be the only access method. if you need more, you should also be able to get it. but if I only want to apply a simple transform, then my plugin doesn't need to know more..

(surely CommentPlugin is and example that only needs a get and a transform)

ok, ok i agree, i had another dumb idea

-- SvenDowideit - 02 Mar 2004

I am in line with what John stated above.

Reason for deprecating the readTopic function: I guess the art and difficulty of defining a good Plugin API. We should avoid exposing implementation details of TWiki in the Plugins API because it makes it more difficult to refactor the TWiki internals over time. The meta data class is one example.

The correct approach is to provide functions to manipulate text received by readTopicText, e.g. to split meta data and topic text, to manipulate meta data, and to merge it back. This has been discussed elsewhere on Codev.

-- PeterThoeny - 10 Mar 2004

I'm sorry, Peter, but that argument just doesn't work. By removing readTopic you have achieved the exact reverse of your stated goal. Without this function, plugins authors are forced to know that meta-data is stored interleaved in topics, and are forced to handle meta-data themselves. To me, the very act of exposing the existence of meta-data is a fundamental breach of the idea of implementation hiding. The fact that authors of new plugins have chosen to continue to use readTopic even after it was deprecated indicates that they understand this.

I completely disagree with you over the "correct approach". You have already provided the seed of an excellent API in readTopic, unfortunately now deprecated, that shelters the plugin author from the need to know that meta-data is stored interleaved in text. Why on earth do you now want to break that? Your proposal exposes more implementation detail than I care to think about. It makes far more sense to publish an API to a meta-data object that the plugin author can talk to.

I am not advocating opening up TWiki internals to plugins. In fact, I'm trying to argue for the exact reverse of this. If you don't publish interfaces to those pieces that plugins authors have to use - such as meta-data - then they are forced to call the internals. Not publishing interfaces is self-defeating. Consider AnalysisOfPluginsAPIUsage; count the number of functions/variables not published by the API that are in use in Plugins today. Nearly 90. And most plugins authors jump through hoops to avoid calling non-API functions. I have personally re-coded plugins three or four times to try and find ways around calling internal functions. You are already exposing the internals to plugins, because you have forced plugins authors into short-circuiting the API.

I fully accept that much of the TWiki internals is too messy/complex/undocumented to publish today. But historically, the amount of refactoring work that has been done to address this has been absolutely minimal. The issue with readTopic was highlighted over a year ago and nothing has been done. All that was required was the publication of a minimal, clean, simple interface to meta.

The only benefit to be gained from the current strategy is that it lets you say "I told you so" when all the plugins break due to refactorings. It piles all the pressure back onto the plugins authors. As a plugin author I'm starting to see the relevance of AndreaSterbini's quote from Dante now; it is indeed like being in the fourth circle.

-- CrawfordCurrie - 10 Mar 2004

I agree with Crawford - we are better off reducing the amount of information that we expose to the plugins, by making Store & Func seperate the types of info (and re-combine them). This will help when the data gets stored in non-linear data stores (like a database..)

so i think we should have

• get/set text
• get/set twiki forms data (maybe for a particular form)
• get/set real meta data
  • revisions info
  • permissions info
  • ...

-- SvenDowideit - 10 Mar 2004

I also agree with Crawford's point.

-- MartinCleaver - 28 Apr 2004

It would be interesting to have the following set of methods:

• readTopic($web,$topic) -> ($text,$meta)
• readTopicText($web,$topic) -> $text
• readTopicMetaData($web,$topic) -> $meta
• saveTopic($web,$topic,$text,$meta)
• saveTopicText($web,$topic,$text)
• saveTopicMetaData($web,$topic,$meta)

You can't isolate the plugins from the fact that there's some metadata associated with the topic. The best course of actions would be to provide a nice API to manipulate that metadata. If that means to publish the TWiki::Meta, fine. If that means to create a TWiki::Plugins::Meta class to serve as frontend to TWiki::Meta, fine. But something needs to be done, as the Metadata mechanism will give a lot of power to the plugins (for example, a WorkflowPlugin could be created that stores the topic state in metadata and changes the form appropiatelu, without hacking the Core like the )WorkflowAddOn).

Meta-Data handling functions and the "OO way" for plugins are IMO the most important things to ne done to easier the plugin creation.

-- RafaelAlvarez - 24 Aug 2004

For a (another) implementation of OO plugin handling, check PluginOOApi . Now that it's (partially) implemented, and tested and measured, I have to ask myself and to everyone: Was it worth the effort? Is really a OO way to build plugins that useful? I saw some small (if any) performance increases, and several (small) performace degradations. The model is not that different to the TWiki::Func interface, so perhaps is not the way. Perhaps something more "profound" is needed.

OTOH, a base plugin can serve as a Facade to all the TWiki::Contrib modules, providing stubs if the proper package is not installed.

-- RafaelAlvarez - 08 Sep 2004

I've looked hard at accelerating plugins, and it basically requires some heavy refactoring of the core rendering loop to achieve performance improvmenets. Because the plugins API is so tightly tied to this architecture, that implies a huge number of changes to plugins. In the short term, plugins can take advantage of the registeragHandler method in TWiki.pm for considerable improvements in tag handling performance, but until there are specific proposals accompaied by a commitment to upgrade existing plugins there is no point in scheduling anything else. -- CrawfordCurrie - 15 Feb 2005