Feature Proposal: So you want to write a plugin

Motivation

Share my experience and setup with others to make life a little easier

Description and Documentation

Some documentation and experience in writing a plugin/REST handler. Includes a directory structure and a makefile

Introduction

Writing a plugin is not easy. the documentation is scant and there is no spec for a good setup.

I have struggled with this for a while and found a satisfactore way of working. Thought I share it here for others to improve.

I am neither a regular perl programmer, nor a makefile expert. However, I find both tools convenient and well documented. If you have better ways of doing things, feel free to share them.

My particular struggle started with the 4.0 upgrade. we used a customised add on that failed in the upgrade. I tried to rewrite adn ran into a variety of problems. Finally SvenDowideit suggested I write a REST handler (See: TWikiFuncAndTheRestInterface). I could not get something that worked then. But I now have a solution and written a development environment.

Development environment

I mostly work on my laptop. It does not run a web server, but it does have a rudimentary, poorly maintained twiki environment. The directory structure is:

twiki402
	bin
	data
	development
	pub
	templates

In addition to the standard directories I have a development directory, where all the new code is stored. Within the development direcory I create

• a makefile
• the plugin I am developing
• Whatever else I need for development. test, etc.

	development
		makefile
		NewPlugin.pm

All testing is done on a target machine. The targets vary, depending where I am working. This is done via the makefile, using:

make install
make test

install updates the required sources on the target machine using a remote copy. For instance:

scp NewPlugin.pm userid@192.168.1.162:/home/httpd/twiki402/lib/TWiki/Plugins/NewPlugin.pm

test executes the development in the target environment:

ssh userid@192.168.1.162 'cd /home/httpd/twiki402/bin; sudo -u nobody ./rest NewPlugin.new -p1 param1 '

test has proven invaluable, because I can actually run from the commandline as the webuser nobody. Prevents a lot of issues, when you are trying to do filesystem manipulations.

Getting started

To set up for plugin development, you need to:

• copy the EmptyPlugin.pm source to your NewPlugin.pm in the development directory
• change all references inside NewPlugin.pm from EmptyPlugin to NewPlugin
• configure the plugin in lib/LocalSite.cfg. You can use bin/configure to do this through the browser, once you have created the plugin.
• create a topic NewPlugin for the plugin in the TWiki web

And if you plan to use the REST interface you must:

• change EmptyPlugin.pm line 137: TWiki::Func::registerRESTHandler('example', \&restExample); to register your resthandler to (for instance)
  TWiki::Func::registerRESTHandler('new', \&restnew);
• change EmptyPlugin.pm line 666: sub restExample { to (for instance) sub restnew {

Once you got this done, you are able to see your plugin operate throug:

• from the command line: make install; make test
• from the browser through: http://192.168.1.162/twiki402/bin/rest/NewPlugin/new

The complete makefile

The makefile as I used it for the replacement of the add on follows:

# run the NewPlugin.new script
SCRIPT = new
PLUGIN = NewPlugin

# the parameters for the call to the REST handler
# http://192.168.1.162/twiki402/bin/rest/NewPlugin/new?method=web&major=4&minor=0&inclusions=.*&exclusions=Web.*&theweb=Rollout&publishdirname=publish&extension=htm
THEWEB = -theweb Rollout
THEMETHOD = -method web
THERELEASE = -major 4 -minor 0
DEBUG = -debug 1
PUBLISHDIRNAME = -publishdirname publish
EXTENSION = -extension htm

PARAMETERS = ${THEWEB} ${THEMETHOD} ${THERELEASE} ${DEBUG} ${PUBLISHDIRNAME} ${EXTENSION}

# a collection of commands to create the plugin on a remote target
TARGET = 192.168.1.162
USERID = userid
WEBUSER = nobody
TARGETDIR = /home/httpd/twiki402
SCOPY = scp ${PLUGIN}.pm ${USERID}@${TARGET}:${TARGETDIR}/lib/TWiki/Plugins/${PLUGIN}.pm
SETPROT = chmod 775 ${PLUGIN}.pm; chown www-data ${PLUGIN}.pm; chgrp www-data ${PLUGIN}.pm
CONFIG = if grep -c ${PLUGIN} LocalSite.cfg; \
           then sed -i -e "s/cfg{Plugins}{${PLUGIN}}{Enabled} = 0;/cfg{Plugins}{${PLUGIN}}{Enabled} = 1;/" LocalSite.cfg ; \
           else sed -i -e "s/^1;/\$$TWiki::cfg{Plugins}{${PLUGIN}}{Enabled} = 1;\n1;/" LocalSite.cfg; fi

# test the script in the target environment from the command line
test:
   ssh ${USERID}@${TARGET}  'cd ${TARGETDIR}/bin; sudo -u ${WEBUSER} ./rest ${PLUGIN}.${SCRIPT} ${PARAMETERS} '
   
# install the modules on server 
install: compile
   ${SCOPY}

# compile the code locally, just in case I made a typo
compile:
   perl -w ${PLUGIN}.pm 

# configure the plugin in the LocalSite.cfg and set the permissions on the plugin   
configure:
   ssh ${USERID}@${TARGET} 'cd ${TARGETDIR}/lib; ${CONFIG}; cd TWiki/Plugins; ${SETPROT}'

# create a plugin with a REST handler from scratch   
new:
   scp ${USERID}@${TARGET}:${TARGETDIR}/lib/TWiki/Plugins/EmptyPlugin.pm ${PLUGIN}.pm
   sed -i -e "s/EmptyPlugin/${PLUGIN}/g;" ${PLUGIN}.pm
   sed -i -e "s/registerRESTHandler('example', \\\&restExample)/registerRESTHandler('${SCRIPT}', \\\\\\&rest${SCRIPT})/" ${PLUGIN}.pm
   sed -i -e "s/sub restExample {/sub rest${SCRIPT} {/" ${PLUGIN}.pm   

A few comments on the makefile

• new will create a working plugin in the development directory. It took a while to figure out the escapes!
• configure creates the appropriate line in LocalSite.cfg, or changes it if it already exists. It also sets the appropriate permissions, owner and group on the plugins.
• compile is there only to save me from from putting pathetic code on the target machine. It does not take much time and saves heaps of frustrations once you have a working version and do most of your tests through the browser.

Some observations

The rewrite of the plugin and the use of the rest handler were both prompted by the upgrade to TWiki 4.0, which broke the previous add on. The rewrite aimed to remove all use of undocumented API features. Looking over the code, I did achieve that.

With the exception of one use of TWiki::Prefs. After I have put the TWiki topic in the appropriate template, I need to insert the following code so that the variable expansion and rendering to work:

 $page =~ s!%TEXT%!$topictext!;
  
      # SMELL: need a new prefs object for each topic (BIZARRE: from Publish.pm)
      my $twiki = $TWiki::Plugins::SESSION;
      $twiki->{prefs} = new TWiki::Prefs($twiki);
      $twiki->{prefs}->pushGlobalPreferences();
      $twiki->{prefs}->pushPreferences($TWiki::cfg{UsersWebName}, $wikiName, 'USER '.$wikiName);
      $twiki->{prefs}->pushWebPreferences($prefs{'web'});
      $twiki->{prefs}->pushPreferences($prefs{'web'}, $topic, 'TOPIC');
      $twiki->{prefs}->pushPreferenceValues('SESSION', $twiki->{client}->getSessionValues());

    
  $page = TWiki::Func::expandCommonVariables($page, $topic, $prefs{'web'});
  $page = TWiki::Func::renderText($page, $prefs{'web'});

I borrowed that code from an older version of PublishContrib. That author did not think much of it either. I am not familiar enough with the TWiki internals to know what causes this, and hence unable to correct it. I noticed an alternative in the latest PublishContrib by SvenDowideit. Bu since i am unclear about cause, impact and side effects I have left as is.

On a positive note, I used the global variable our %prefs; to carry the list of parameters used in the various routines. I saw this in the GenPDF plugin. Thanks for the idea. it makes for much more readable code and separates the variables that matter clearly from the constants.

And one more puzzle. The rest handler is passed a session variable. Is there any documentation on how to use this variable. To paraphrase an old haiku:

A variable that important,

it must be usefull.

But I can't use it.

Thanks for TWiki. It has now served us for five years in the maintenance of a large static web side. two releases a year. reliable, flexible and easy to use.

Examples

Impact

Implementation

-- Contributors: BramVanOosterhout - 17 Jun 2007

Discussion

It's great that you have shared this, Bram. Any particular reason that you don't use the BuildContrib? It's the recommended tool for extension developers, as it automates almost everything your makefile does in a five-line build.pl, as well providing the create_new_extension.pl script which simplifies new plugin creation. It also integrates closely with the TWiki unit testing methodology, and the configure - based extension installer.

I don't know what you add-on does, but I assume from your use of that code that you need to change the web/topic context. (the purpose of the bizarre code is to set up the preferences environment for a specific web/topic pair). 4.2 has the pushTopicContext method, which effectively makes the code you used part of the official APIL

=pod

pushTopicContext($web, $topic)

• $web - new web
• $topic - new topic

Change the TWiki context so it behaves as if it was processing $web.$topic from now on. All the preferences will be reset to those of the new topic. Note that if the new topic is not readable by the logged in user due to access control considerations, there will not be an exception. It is the duty of the caller to check access permissions before changing the topic.

It is the duty of the caller to restore the original context by calling popTopicContext.

Note that this call does not re-initialise plugins, so if you have used global variables to remember the web and topic in initPlugin, then those values will be unchanged.

Since: TWiki::Plugins::VERSION 1.2

=cut

On the session variable; this is a pointer to the TWiki object. It is the same object pointed to by $TWiki::Plugins::SESSION= during plugin execution. It is useful - incredibly useful - but in the interests of portability you should avoid using it at all costs. It is passed to REST handlers because there are some REST functions that simply can't manage without it :-(.

-- CrawfordCurrie - 17 Jun 2007

Thanks Crawford.

Re: Any particular reason that you don't use the BuildContrib?

Simple, I did not know it existed. And, now that i have read the doco, I don't see immediately how it helps with what i wanted to do.

As I said, I am a complete novice, who knows some perl and likes makefiles.

I think the BuildContrib advice is in the same leage as SvenDowideit advice on the REST handler. An excellent idea, if only I knew how.

I will try next time I need to write something.

Thanks for the suggestion.

Re: $TWiki::Plugins::SESSION

I think my ignorance is bliss! I will avoid/ignore it.

-- BramVanOosterhout - 22 Jun 2007

When I tried to implement the above solution in TWiki 4.1.2, it failed. See CantCallMethodGetSessionValues.

I reread this note and attempted to implement the SMELL code above by the latest in PublishContrib. Like:

######### Replace this lot by the latest implementation in PublishContrib  
#     # SMELL: need a new prefs object for each topic
#      my $wikiName = 'TWikiGuest';
#    my $twiki = $TWiki::Plugins::SESSION;
#    $twiki->{prefs} = new TWiki::Prefs($twiki);
#    $twiki->{prefs}->pushGlobalPreferences();
#    $twiki->{prefs}->pushPreferences($TWiki::cfg{UsersWebName}, $wikiName, 'USER '.$wikiName);
#    $twiki->{prefs}->pushWebPreferences($prefs{'web'});
#    $twiki->{prefs}->pushPreferences($prefs{'web'}, $prefs{'topic'}, 'TOPIC');
#    $twiki->{prefs}->pushPreferenceValues('SESSION', $twiki->{client}->getSessionValues());
#---------------------------------------------------
    # clone the current session
    my $oldTWiki = $TWiki::Plugins::SESSION;

    # Create a new TWiki so that the contexts are correct. This is really,
    # really inefficient, but is essential at the moment to maintain correct
    # prefs
    my $query = $oldTWiki->{cgiQuery};
    $query->param('topic', "$prefs{'web'}.$topic");
    my $twiki = new TWiki('TWikiGuest', $query);
    $TWiki::Plugins::SESSION = $twiki;
##########   end replace
##########   note the restore twiki object four lines down  


  $page = TWiki::Func::expandCommonVariables($page, $topic, $prefs{'web'});
  $page = TWiki::Func::renderText($page, $prefs{'web'});
# do it twice, in case the rendering defines more variables
  $page = TWiki::Func::expandCommonVariables($page, $topic, $prefs{'web'});
  $page = TWiki::Func::renderText($page, $prefs{'web'});
  
  $TWiki::Plugins::SESSION = $oldTWiki; # restore twiki object

And now the script works again. As Sven notes, the solution is not fast: 4 minutes to render 800 pages (0.3 second per page). It was under .1s/page in 4.0.2. But it works. And, I have achieved my objective to eliminate all dependencies on undocumented API calls. I hope this is the end of my incompatibility problems!

Thanks to SvenDowideit , for putting the solution together.

-- BramVanOosterhout - 03 Jul 2007