The MathematicsWiki add-on is a perfect example of extending the TWiki syntax. LaTeX rendering is not something everybody needs, but some TWiki installations probably depend on it. Therefore this should not go into the TWikiProductionRelease, it is a TWikiPlugin. We could enhance TWiki so that TWiki extensions can be discovered automatically, i.e. offer an API for TWiki add-ons.

Currently the TWiki syntax can be extended by editing the wikicfg.pm, there are the functions extendHandleCommonTags, extendGetRenderedVersionOutsidePRE and extendGetRenderedVersionInsidePRE that allow you to do that. Upgrading is eased because ideally you only have to merge changes in wikicfg.pm. This is not always possible in practice however. Some extensions require changes in other files as well.

We could bring that to the next level, an API so that TWiki can discover new rules on the fly. This is called introspection in JavaBeans talk.

Idea how it could be done:

• The /twiki/bin/plugin directory contains TWikiPlugins.
• It contains add-on scripts, i.e. latex.pm would handle the LaTeX rendering.
• Each plugin is a script that needs to implement a set of functions. Example:
  • sub initialize
  • sub handleCommonTags
  • sub getRenderedVersionOutsidePRE
  • sub getRenderedVersionInsidePRE
• TWiki reads all scripts and calls the relevant function of all add-on scripts.

Adding an add-on to TWiki is simply a matter of copying a file into the /twiki/bin/plugin directory.

For performance reasons it is best to use "call by reference" parameters instead of "call by value". Example call:
handleCommonTags( $text );

Example function using "call by reference":

sub handleAllPrefsValues
{
    # do text manipulation directly
    $_[0] =~ s/fromFoo/toBar/go;
    # no need to return value
}

This is now in procedural speak, it could be done in OO (see HowShouldTWikiBeModularized).

Questions:

• What other functions of the main TWiki engine should be exposed, e.g. offered a hook?
• Is there a need to offer hooks into the templates? If yes, how?
• How to handle configurations if needed? Ideally that would be a TWiki topic, similar to the TWiki preferences topics.

On a higher level, we should define how to package TWiki plugins. I.e. a ZIP file containing all required files (plugin script, help topics, configuration topics, template files) in a recommended directory structure.

Should installation also be automated? An "Add-a-TWiki-Plugin" topic and script could do that. The topic could show the officially available add-ons (polled from the TWikiPlugin topic in TWiki.Codev web) and install a selected plugin automatically on a target system. The plugin ZIP file probably needs also a config file for the installer, i.e. required TWiki version and Perl version. The question of security arises, installing executable scripts is a scary thing.

-- PeterThoeny - 26 Sep 2000

TWikiPlugin modules should be perl modules installed under say TWiki::Plugin. This gives:

• LIB/use lib syntax for module code discovery
• BEGIN/END code blocks
• perl Makefile.PL installation process
• perl version pacakage maintaince
• perldoc
• Better code respoitory management.

Then in the usual TWiki way activation and configuration via a Topic. This of course requires some access control. A eval block would be used to test existance, and avoid initialization errors breaking the server.

Note, I expect that currently the cost of starting a perl process far out weights the difference between passing a scalar reference and string averaging less than 8Kb.

-- NicholasLee - 26 Sep 2000

I would add to the above feature list also:

• a testing framework (let do ExtremeProgramming !!!)

I would like to extend the TWikiPluginAPI proposal. As a Wiki user, my dream is to be able to integrate features implemented in different Wikis without having to adapt programs.

Several (perl?) Wikis are available around, each one featuring some very interesting feature: (I list just some I remind now, in no particular order of appreciation)

• Ward's LiterateDevelopement, SimilarTopicNames, .... (too many to mention)
• TWiki you know
• APW an efficient caching engine ...
• ModMod new ways to expose the WikiIceberg (DesiredPages, ...)
• Kehei A PlugInProtocol ...
• WebWebX Web management ...
• ...

We could port the features we like to TWiki ...

[ I would prefer a common API between wiki implementations, both for plugins and for topic format ... I hope our API will be clean enough to attract other wiki developers ]

-- AndreaSterbini - 27 Sep 2000

I think that the simplest way of configuring twiki for the "casual" administrator is through wiki itself: in the WebPreferences or a similar page one should also put the code for the definition (*SET VARIABLE = somethiong) of the %VARIABLES/functions (clearly, this page should be protected) In this way extensions/customization is simply a matter of copying text.

Moreover, for external plugin like LaTeX or gnuplot, I would suggest also extending the %INCLUDE function:

• INCLUDE/SET VARIABLE = url{http://url} includes/set a variable to the html of the url,
• INCLUDE/SET VARIABLE = shell{command} includes/set a variable to the output of command
• INCLUDE/SET VARIABLE = perl{perl fragment} includes/set a variable to the output of the perl fragment
• INCLUDE/SET VARIABLE = wiki{sometext.txt} includes/set a variable to the content of a text file intepreted a' la wiki
• INCLUDE/SET VARIABLE = text{sometext.txt} includes/set a variable to the pure content of a text file

Obvoiusly, for security reasons, one should state into the WebPreferences the "security patterns" to which the commands have to be subitted before execution, and the script itself could chroot before execution.

Examples of applications:

1. INCLUDE url{http:/cgi-bin/latex2html?expr=$x%3D\int_3^4g(y)dy$}
2. INCLUDE shell{/usr/local/bin/latex2html <<END; $x=\int_3^4 g(y)dy$;END}
3. SET A = perl{%B%+%C%}

The first option allow every "pluging" to be installed if a suitable CGI interface is developed, the second is quickier (non need to pass trough CGI-perl-external program) but more dangerous, the last option can be used to produce "computed" text like for instance multiple choice quizzes that depend on some parameter that changes randomly at every run.

-- FrancoBagnoli - 29 Sep 2000

Interesting idea Franco! Probably not too difficult to implement. Grabbing a URL is already implemented by the geturl TWiki script.

But security is a big issue. If implemented, a $securityLevel variable should be added in wikicfg.pm that defines the level of security.

It could also be protected by allowing execution only if the topic has been saved by a person in the TWikiAdminGroup.

-- PeterThoeny - 01 Oct 2000

I am tired of patching last alpha release to add all the Twiki extensions I like (FrancoBagnoli's LaTeX and Reply, my webnames, TOC, subwebs, and get-a-web, ...).

I am implementing a simple plugin package:

[ description superseeded by my implementation below ] -- AndreaSterbini - 20 Oct 2000

-- AndreaSterbini - 06 Oct 2000

It might be worth creating a new cvs module to deal with plugin development.

How about the OO way:

   my @modules = qw/Latex Subweb/;  #not using Toc
   my @plugins = ();
   foreach $name (@modules) {
      use TWiki::Plugin::$name; 
      my $plugin = new TWiki::Plugin::$name(  "some stuff" , ... );
      $plugin->init(...);
      # Time to register what the plug provides?
      push @plugins, $plugin;
   }

   foreach (@plugins)
   {
     # Probably worth downing to existance testing here.
     $_->handleMyCommonTag( ... );
   }

Would has about the cost as the above, plus its much easier to maintain. Ideally a TWikiPluginAPI would provide a subclass for this sort of thing.

Note: the -T taint warning is probably got to do with the '$name' variable.

-- NicholasLee - 06 Oct 2000

Procedural or OO, either way is fine with me. It is important to make it transparent for the developer and user.

I would go for a fixed subdir below twiki/bin, i.e. twiki/bin/plugin. The name of the subdir can be defined as a (static variable in wikicfg.pm. Installing a new plugin is simply a matter of copying the script into this directory. The init function could scan the plugin directory and install all script files at init time.

If you want to have control over the sequence of loading or over enabling/disabling the plugins you could create a PluginConfig topic, or better yet, simply a new TWikiPreferences variable, i.e.

• Set PLUGINS = latex.pm, subweb.pm

We should move the extend* functions from the wikicfg.pm file into a plugin/default.pm file. This file can be used by developers as a template file to create new plugins.

BTW, the TOC functionality should go into the distribution as soon as we decide on the syntax and the code is ready.

-- PeterThoeny - 07 Oct 2000

This weekend I have tried a simple implementation based on packages. I attach here my files.

• plugins are in .../twiki/plugins (you can change it)
• each one is a package that defines the 4 subs:
  • initPlugin (query, text, web, topic)
  • commonTagsHandler (query, text, web, topic)
  • outsidePREHandler (query, text, web, topic)
  • insidePREHandler (query, text, web, topic)

• I load wikiplugins.pm (here the plugins are configured) and call

   applyPlugins( "initPlugin" , $query, $text, $web, $topic );

• in each extendXXX sub I call

   applyPlugins( "commonXXXHandler" , $query, $text, $web, $topic );

It's simple, it's easy, and it can be improved a lot :-).

I enclose here also 6 plugins:

• dummyPlugin (just a template)
• webname (to render multilevel web names and to show web.WebHome as web )
• toc (for the TOC tag and for H1-H6 SectionTitles)
• latex (see FrancoBagnoli LaTeX interface to hevea )
• reply (see FrancoBagnoli threaded topics)
• getaweb (to replace web.zip with a link to the GetAWebAddOn web zipper)

NOTE (bug?) all extendXXX routines in wikicfg.pm should end with return $text; instead than with return $_ otherwise we loose all the good work done on $text .

[ FIXME: perhaps I have unwrapped a parameter list somewhere ... ]

-- AndreaSterbini - 08 Oct 2000

I Changed extensively the latex,gnuplot, reply, etc. plugins.

I'm uploading all my bin directory as an attachment in WebTeach, and I'll try to adhere to Sterbini's standards in the next version.

[ my "standards" ... ... they are already changing ... see below ] -- AndreaSterbini - 20 Oct 2000

-- FrancoBagnoli - 09 Oct 2000

I have updated my plugins (GPL license added) and made a first porting of ModMod InterWiki feature as a plugin.

I have noticed that commonTagsHandler is called both on the page template and on the topic text ... this means that tags inside <PRE> areas will be transformed ... Is it this the correct behaviour? I would prefer that:

• outsidePREHandler is used only on topic text
• insidePREHandles is used only on topic text
• commonTagsHandler is used only on templates (if we need its services on the text topic we add rendering rules to outsidePREHandler

Notice that outsidePREHandler/insidePREHandler are called on each line of text, this has two drawbacks:

• if we want to catch multi-line patterns (see latex plugin) we are forced to use commonTagsHandler
• it's less efficient

-- AndreaSterbini - 14 Oct 2000

I agree with NicholasLee's suggestion of using OOP, as soon I learn how OOP is done in Perl I give it a try.

For the moment, I have added an if defined(&$sub) in the applyPlugin code, so that I can forget about non-defined handlers.

-- AndreaSterbini - 16 Oct 2000

Hi, TWikiers!

I am packaging my small plugins (smilies included ). Each plugin is packaged as a zip file, e.g. named SmiliesPlugin.zip that contains:

• plugins/SmiliesPlugin.pm (the code)
• data/TWiki/SmiliesPlugin.txt (documentation, installation instructions and configuration settings)
• pub/TWiki/SmiliesPlugin/smile.gif ... (all the needed files)

This way the installation is reduced to:

• download and unzip the file in the twiki installation directory
• add "SmiliesPlugin" to the ACTIVEPLUGINS variable in TWikiPreferences (comma-separated list)

I am using the zip file name both for the documentation and for the perl package name so that all links automagically to the ACTIVEPLUGINS list (this is the "true wiki way" ).

I have played a little with the OOP code proposed by NicholasLee. Unfortunately my perl chokes on the use $pluginName; line ... from the manual I see that this line is the same than a BEGIN block ...

[ I am now using =eval "use $plugin;" ... it's the only way ...]

I am considering a change to wikiplugins.pm :

• read the list of active plugins from the ACTIVEPLUGINS preference variable This way the administrator could deactivate plugins without editing the code.
• learn to use perl autoload
• learn the perl OOP way.

I agree, we should open a CVS tag for the plugins developement, can I?

-- AndreaSterbini - 20 Oct 2000

Nice stuff Andrea, TWiki gets much more flexibel! Very busy at the moment, not yet studied your code much. Nevertheless, a few comments.

• Each plugin can (but does not need to) have the following functions:
  • initializePlugin
  • commonTagsHandler
  • outsidePreHandler
  • insidePreHandler
• Add an initializePlugin function in wikiplugins.pm, which is called at the end of initialize. It does:
  • Initialize the plugin handler itself (for example, read active plugins from a topic as you described. Or better, read all but the plugins in a disabled plugin list?)
  • Build lists of file names, one each for initializePlugin, commonTagsHandler, outsidePreHandler, insidePreHandler. Each list has the plugin filenames where a function exists, i.e. the @filesWithInitializePlugin list has all files where initializePlugin exists. This is for performance, so that we do not have to check this later on.
  • Call the initializePlugIn function on all plugins so that they can initialize.
• File wikicfg.pm: Better to remove the functions extendHandleCommonTags, extendGetRenderedVersionOutsidePRE, extendGetRenderedVersionOutsidePRE and move them into a default plugin. They should have the same function names as the plugins and use the same "call by reference".
• applyPlugins should be called from the relevant places within wiki.pm, not wikicfg.pm.
• What is the first parameter ($query) in each function, is it necessary?

That's it for now. More later.

-- PeterThoeny - 21 Oct 2000

I have just uploaded the new version of my plugins, all packaged.

There are two new entries:

• SmiliesPlugin, a plugin made to handle SmiliesVariables.
• InterwikiPluginEarlyDev, a porting of the Interwiki feature of ModMod Wiki.

To install plugins:

• add wikiplugins.pm to the twiki/bin directory
• change wiki.pm and wikicfg.pm to handle plugin initialization
• install ALWAYS DefaultPlugin.zip if you comment the subs in wikicfg.pm
• in wikiplugins.pm put the correct location of the plugin directory (I will fix this later)

• download the XXX.zip file
• unzip it in the twiki installation directory
• add to TWikiPreferences the variable
  • Set ACTIVEPLUGINS = DefaultPlugin, TocPlugin ....

TODO:

• implement the suggested handler's registration to improve efficiency
• use perl Autoload or Selfload to load code only if used by the handlers (i.e., if a rule does'nt match then the plugin support code is not even loaded/compiled)
• use compiled patterns where possible

NOTE: you can forget all the old =*.pm* attachments below except wikicfg.pm, wiki.pm, wikiplugins.pm.

Re: About the parameters ... the current implementation passes at most 4 parameters to each handler ... I think $query will be very useful for the SessionVariables plugin I am implementing. If you prefer I move it to last position so that first argument is always $text.

PS: Next, when the API will be part of the core code, all the plugins attached here should be moved to the TWikiPlugins topic.

PPS: about the *.pm plugins and files ... for security reasons I suggest we make a lib directory outside the bin directory to contain all of them (this way we could stop minding if somebody points his/her browser to .../twiki/bin/wiki.pm or other module).

-- AndreaSterbini - 24 Oct 2000

To keep each handler simple and improve efficiency, I am changing the parameters of the handlers explained above as follows:

• initPlugin( $web, $topic, $user )
• commonTagsHandler( $text )
• outsidePreHandler( $text )
• insidePreHandler( $text )

• beforeStoreHandler( $text )

Comments?

[I have not yet uploaded the new plugin versions ...]

-- AndreaSterbini - 31 Oct 2000

The $theWeb and $theTopic parameters need to remain in the parameter list because some places like search call it with a different web or topic then at init time. It is fast since it is only a reference that is passed.

For initPlugin I recommend to use the same parameters as &wiki::initialize is returning, e.g.

• initPlugin( $theTopicName, $theWebName, $theScriptUrlPath, $theUserName, $theDataDir );

Good idea about the hook before storing the topic. I suggest to do that with the text data from the edit form, so that preview does reflect the plugin rendering as well.

• afterTopicEditHandler( $theText, $theWeb, $theTopic );

Complete list:

• initPlugin( $theTopicName, $theWebName, $theScriptUrlPath, $theUserName, $theDataDir );
• commonTagsHandler( $theText, $theWeb, $theTopic );
• outsidePreHandler( $theText, $theWeb, $theTopic );
• insidePreHandler( $theText, $theWeb, $theTopic );
• afterTopicEditHandler( $theText, $theWeb, $theTopic );

-- PeterThoeny - 01 Nov 2000

I have just updated the plugin code below and the plugins. Now the plugins initialization "registers" all known and defined handlers of the active plugins in a hash table for efficiency.

The handlers registered now are:

• initPlugin
• commonTagsHandler
• outsidePREHandler
• insidePREHandler
• beforeSaveHandler
• afterEditHandler

I have not yet used the above argument lists ... see you at next episode

-- AndreaSterbini - 07 Nov 2000

Just a note ... I have found some bugs in the ReplyPlugin ... please wait for an upgrade.

-- AndreaSterbini - 18 Jan 2001

I've added a first cut of the Drawing Applet as an add-in to TWikiDrawImageMaps, it also includes the ability to produce client side image maps.

-- JohnTalintyre - 19 Jan 2001

Questions to Andrea:

• Are the attached *.pm files the latest ones? I will soon update the TWiki core.
• Is sub emitTR in wikicfg.pm needed? I'd prefer to get rid of code in wikicfg.pm.

Note to Andrea and myself: We need two more hooks to track state in the getRenderedVersion function (see TableExtensionProposal) :

• getRenderedVersionInit to handle customized initialization of getRenderedVersion. Called just before looping through all lines.
• getRenderedVersionCleanup to handle customized cleanup of getRenderedVersion. Called just after looping through all lines.

-- PeterThoeny - 27 Jan 2001

Hi, dear TWikiers.

Yesterday I was looking to some nice CPAN Perl module and I have found time to create a CalendarPlugin. It introduces the tag %CALENDAR% that shows a monthly calendar with days highlighted depending on a list of events defined in some other topic. The complete package is attached below ... just umpack it and add CalendarPlugin to your ACTIVEPLUGINS in TWikiPreferences. (you need also at least HTML::CalendarMonth HTML::ExtendedTable HTML::Parser and HTML::AsSubs from http://www.cpan.org)

Reply to Peter:

• I have upgraded all files below
• emitTR in wikicfg.pm is part (with other 3 lines in wiki.pm) of my ExtendingTableSyntax old idea
• to add new hooks just add new values to the @registrableHandlers variable in wikiplugins.pm (remember to set the LIB path when you install it)

PS sorry for my lazyness ... I keep attaching plugins here instead than to TWikiAvailablePlugins ...

PPS ReplyPlugin is now corrected ... (for a better LatexPlugin wait a little ... FrancoBagnoli is doing something a lot better) ...

-- AndreaSterbini - 27 Jan 2001

I've just attached a HideaccessPlugin that hides the Set ALLOWTOPICCHANGE = setting from all pages.

Moreover, I have moved $pluginsDir variable to wikicfg.pm and made some small (cosmetic?) changes here and there.

There are a copuple of (useful?) subs in wikicfg.pm to retrieve all topics of a web or all web names.

Question: When do we start a CVS branch for the Plugins developement?

-- AndreaSterbini - 29 Jan 2001

Small refactoring to wikiplugins.pm ... (needed by AutomaticLinkSuggestionPlugin )

-- AndreaSterbini - 31 Jan 2001

Probably it would be better if the wikiplugins.pm code reads the ACTIVEPLUGINS from the current properties. This way:

• if you want the old site-wide configuration just add ACTIVEPLUGINS to the list of FIXEDPREFERENCES of TWikiPreferences
• if you want a web-wide configuration set ACTIVEPLUGINS in each WebPreferences and add "ACTIVEPLUGINS" to the FIXEDPREFERENCES of WebPreferences
• if you want user-configurable plugins leave the variable free to the users

Comments?

Proposal we probably need a way to ADD values to a variable, so that you can fix a minimal set of site-wide ACTIVEPLUGINS, each web can add other plugins, the user can add even more plugins. Would you like a syntax like:

   * Add ACTIVEPLUGINS = AnotherPlugin, EvenMorePlugin

-- AndreaSterbini - 01 Feb 2001

This topic is geeting big, lets split it up.

Regarding "CVS branch for the Plugins developement", until we have a clear plan it is better to attach plugins to this topic here. Lets follow up in TWikiPluginDevAndDeployment.

Regarding "Add ACTIVEPLUGINS" feature: Nice idea, follow up in AddValuesToPreferenceVariable.

I will soon update TWikiAlphaRelease with Andrea's first plugin version.

-- PeterThoeny - 06 Feb 2001

Changed initPlugins parameters, now is:

• initPlugin($topic, $web, $user, $installWeb).

The code in latest alpha now accepts only <web>.<name>Plugin in the ACTIVEPLUGINS variable (the rest is ignored). Why?

• the init code does'nt know from which web each plugin in the ACTIVEPLUGINS list is installed (among TWiki, current web or Main web)
• Some plugin need to know where it's installed (e.g. plugins with attached files)
• if we use the syntax "Set ACTIVEPLUGINS = FirstPlugin, %ACTIVEPLUGINS%, AnotherPlugin" then all links in each preference topic is correct

I forgot to say that I am (slowly) uploading all my plugins in the new Plugins web ... (see WebHome )

-- AndreaSterbini - 04 Mar 2001

One pending documentation is OfficialCoreSubsForPlugins.

-- PeterThoeny - 12 Mar 2001

I propose all modules have a VERSION variable to let all plugins check if thay can be properly initialized. See PluginAPIGetVersion for a slightly longer description.

-- AndreaSterbini - 27 Mar 2001

The present TWikiPluginAPI is mainly about the data files, mainly covering rendering to HTML. At my company we've changed some key routines to do user identification, partly done by using Session cookies. I would like to propose some extra plugin capabilities for this - TWikiPluginAPIForIdentification.

-- JohnTalintyre - 03 Apr 2001

It would be useful if an extra parameter was passed to outsidePREHandler, that is: noautolink. The alternative is for the code that checks for this to be replicated in plugins.

-- JohnTalintyre - 04 May 2001

I have implemented and uploaded in last alpha the following small changes:

• the initPlugin function should return 1 if the initialization went OK, or else 0 to signal that the plugin should NOT be registered.
  • THIS BREAKS CURRENT PLUGINS: THEY SHOULD BE UPDATED
• how the installationweb of a plugin is determined:
  • if the plugin is named <web>.SomePlugin then use <web>
  • else look in TWiki for the SomePlugin topic
  • else look in current web for the SomePlugin topic
  • else look in Main web for the SomePlugin topic

• now Plugins.pm and default plugins contains the variable $VERSION='1.000'

-- AndreaSterbini - 29 May 2001

I am working on a Peer Review plugin (see PeerRatingSystem). There are a couple of things that I would like to see added to the Plugin API:

• I am using TWiki::formatGmTime() and TWiki::Store::readTemplate() directly - should these be served up through the API?
• Access to params (it seems that this is now available because cgiQuery is a global ... but I am not sure that just using globals is in the spirit of the Plugin API)
• Access to %META% data - in particular to topic revision number

Any thoughts on how to do this - I would be happy to code it up if needed.

-- SteveRoe - 21 Jun 2001

I have been trying to get the WebnamesPlugin to work for awhile now, and I finally fixed the problem. I didn't want my Web's to show up as TWiki? when I typed in TWiki.TWiki, so I modified the outsidePREHandler sub to contain:

$_[0] =~ s/(^|\s)([A-Z]+[a-z0-9_\-]*[\.\/])+([A-Z]+[a-z0-9_\-]*(\s|$))/$1$2$3$4/go; Does this work for anyone else?

Let me know

-- JustinHickman - 06 Jul 2001

I made some spec changes as described in SimplerPluginInstallation and OfficialCoreSubsForPlugins.

It is recommended to update the existing plugins, however they do run without changes. In case you don't you will not get the automatic description in TextFormattingRules. Update is recommended because plugins may only use functions in &TWiki::Func module.

-- PeterThoeny - 14 Jul 2001

Added a plugin called SlashdotPlugin. Go to my documentation page to see what it is.

-- JustinHickman - 13 Aug 2001