Refactoring Proposal: The REST implementation is flawed

Motivation

While trying to upgrade some of my exotic extensions to work with the current MAIN branch, I mused about redesigning them to use the REST interface instead of their own executables in bin. Reading the documentation at REST, TWiki.TWikiFuncDotPm and TWiki.TWikiScripts#rest, and first experiments were less than rewarding.

Description

The REST interface needs to be reasonably documented (as has been stated in some of the topics above), and better implemented.

-- HaraldJoerg - 24 Jul 2007

Impact and Available Solutions

Documentation

Examples

There is an example at TWiki.TWikiScripts#rest: If you run the example of the REST handler, you get an Internal Server Error (code 500). This may be sort of a temporary problem because it sort of works in the MAIN branch - but only if after you have explicitly enabled EmptyPlugin in bin/configure. Who would do that in any real TWiki installation?

Implementation

Announcing REST handlers

How do you make a REST handler known to TWiki? Ah - that's easy. TWiki.TWikiFuncDotPm#registerRESTHandler_alias_fn says:

But wait - that would require my extension to be implemented as a Plugin and not as a Contrib. And the invocation of initPlugin would be called in every TWiki request, not just a rest. Every view, every edit will register the REST handler, though there isn't the slightest chance that it is invoked (since it is, as we read, a view, or an edit). Though it doesn't hurt much with regard to measurable performance impact, this is idiotic.

REST handlers should be registered with either the installation of a Contrib / Plugin, or with bin/configure. Or even auto-discovered: The convention says that the subject should be the name of an extension. Why not try to use that extension instead of checking whether it is a Wikiword? Another simple convention could require that the routine would have a special name, say $verb . 'handler', within said extension, instead of being freely selectable. Would easily avoid that some plugin steals another plugin's handlers by registering wrong subjects, wouldn't it? (Yes, you could still steal by overriding the functions while compiling your extension. But that is another story.)

Running REST handlers

Some parameters of a REST handler are reserved: username, topic, endPoint. Only the latter is documented: If endPoint is given, the script will redirect to a view (hardwired) of that topic. The topic parameter will be defaulted for you (not that you've asked for it), and the username parameter should not be used unless TWiki knows how you log in.

The topic and endPoint parameter need to be documented.

Discussion

I have two approaches to writing REST handlers. Call them the "plugin approach" and the "standalone approach". In the plugin approach, I have a plugin that requires a REST interaction - for example, the recent extension to the ActionTrackerPlugin to modify the status of an action from the view, or the edit modes in the EditRowPlugin. In this case, registering the REST handler from the plugin is no problem, as the REST code is all in the plugin module anyway, and as you point out, the cost of registering a REST handler is negligible.

The second approach, the standalone approach, involves writing a REST script in bin. In that script I simply new TWiki and work my evil will. I use this approach occasionally because, as you point out, there is no way to initialise a Contrib. See InitialiseNonPluginExtensions.

I'm not a big fan of auto-discovery, after measuring how expensive it was for plugins, but the approach you describe above is sensible; I use it in another REST dispatcher for a different application, very successfully.

FYI here's a cut down version of the REST dispatcher in my other application:

sub cgi {
    my ($query) = @_; # CGI object

    $query->path_info() =~ m#^/(.*)#; # untaint and extract the handler name
    my $action = $1;

    my $handler = "REST::$action";
    eval "require $handler";
    die "Failed to load handler for '$action' command: $@" if $@;

    # Handlers die on failure, which automatically causes a 500
    # back to the client.
    $handler .= '::handle';
    no strict 'refs';
    &$handler($query);
    use strict 'refs';
}

-- CrawfordCurrie - 25 Jul 2007

The main reason than the handlers must be "registered" is that it restrict the methods that can be called from a plugin module, the secondary (and minor) reason being that it allows to register the rest call with a different name than the method name.

The reason to use run-time registering of rest calls instead of registering them in configure is that it forces an additional step when upgrading the plugin.

Those are the basic assumptions behind the initial implementation of the rest script (which was later totally rewriten, but keeping the same mechanism). If those assumptions are no longer valid, then what is proposed makes a lot of sense.

Additionally, I would suggest to address one of the "flaws" in the original desing that CrawfordCurrie pointed out some time ago (I don't recall where): rest urls are not easily exchangeable with TWiki urls. Meaning that in a rest url the "web" is the module and the "topic" is the method to be invoked, so to pass a name of a topic to work with you need to do some hacks. So having a url of the form /rest/Web/Topic/Module/Action or something like that could be nice.

-- RafaelAlvarez - 25 Jul 2007

btw, the original discussion of the rest script is at RESTfulPluginInterface

-- RafaelAlvarez - 25 Jul 2007

Rafael, many thanks for that pointer - that clarifies many things.

I tend to agree that having a different meaning for PATH_INFO in rest than in any other script is an unnecessary oddity. However, I am afraid that we are stuck with that oddity, since rest is already in use, unless we invent justanotherrest.

I'd still like to challenge the idea of run-time registering, though you got some points. "... restrict the methods that can be called from a plugin module:" This is quite correct, I've missed that. If it would be allowed to jump right into any routine in any module by simply constructing an appropriate "subject" and "verb", then anything (mostly bad things) could happen. Yet run-time registering seems to be a suboptimal solution: How often do you upgrade? How often do you view? Declaring the set of allowed REST methods is a matter of installation and configuration, and not a matter of regular operation. The same argument would be valid for plugin handlers, but only to a lesser extent: Plugin handlers are called during other requests like view or edit, whereas REST handlers are called instead of them.

Another good point from RESTfulPluginInterface is the concept to "leverage the "standard" script infrastructure". My current implementation achieves that with an own script in bin, mimicking the other TWiki scripts. This brings the hazard of a version dependency, which indeed hit me, though not yet on the scripts in bin. However, I thought that the rest script would be a chance to get rid not only from versioning struggle, but also from exactly said TWiki overhead, which I do not need: There are no plugin handlers to call from my script, no need to parse any preference variables (beyond what's in the configuration hash), no need to compile 90% of the codebase.

So back to the drawing board:

• What is the best way to implement extra scripts which want to use TWiki's infrastructure?
  • Mimic any of the current scripts or
  • Write it as a rest invocation?
• What is the best way to implement lightweight extra scripts which want to avoid the burden of TWiki's infrastructure?
  • Mimic any of the current scripts or
  • Write it as a rest invocation?

Given that the use cases you have provided in RESTfulPluginInterface clearly point to using TWiki infrastructure, the lightweight stuff needs to take another road.

The two alternatives regarding TWiki infrastructure map to Crawford's concept of "plugin approach" and "standalone approach": In the plugin approach of EditRowPlugin (thanks for that example!) the TWiki infrastructure is desired, yet a special handler is needed because the "normal" save doesn't do the right thing, and it is implemented as a TWiki rest handler.

A standalone approach might do very different things, perhaps "lightweight" with regard to TWiki infrastructure (think of an AJAX handler invoked by some Javascript), so should probably go elsewhere. Crawford's pointer to InitialiseNonPluginExtensions brings in the point that a handler might need things which are outside TWiki's infrastructure and can therefore not be handled with the current rest script.

So, bottom line: The REST implementation isn't half as flawed as I thought. I have misunderstood its intention, and will probably stick to my own script in bin for a while...

-- HaraldJoerg - 25 Jul 2007

one thing that you can do instead of making justanotherrest, is to ammend the calling of rest via the generic twiki script so it is consistent. Obviously, with the existance of twiki, and paired with a twikiauth we could deprecate the other cgi-scripts, and build in consistency.

Thus you could (I've not checked) already have this, by creating a Contrib that adds a new TWiki::UI module.

-- SvenDowideit - 26 Jul 2007

IIRC (I don't have the code in front of me), the TWiki instance is created in the TWiki::UI module, before the call is delegated to the proper UI module.

BTW, rest used to be a TWIki::UI module, and was later changed to have everything in the script. I missed that discussion and change. Can someone tell me why it was done that way?

Harald, by Lightweight you mean "no plugin initializacion" or "no twiki initialization"?

If it's the later, then the rest script can be made more "intelligent" by having two sets of registered handlers: One for those that use TWiki infrastructure, those that should be stand-alone. And if the handler is not in the first set, then look in the second set and invoke the handler without initializing TWiki.

-- RafaelAlvarez - 26 Jul 2007

Rafael, I moved the code into the bin script for a couple of reasons. First, i wanted it to load only the code that was absolutely necessary, and second, the exception trapping in the UI module is incorrect behaviour for REST. Most rest scripts do not want to get a 302 in the face - much less a 200 when a function fails!

-- CrawfordCurrie - 26 Jul 2007

Calm down, folks! Methinks that you are starting to take this topic more serious than I ever intended.

• I am definitely not going to use the generic twiki script. Read TWiki.TWikiScripts#twiki and you know why. Its designated use cases, consequences on robots.txt (which I am using extensively to control our corporate search engine), and applicability to a ShorterUrlCookbook setup are far from clear to me. If anybody would, at some time in the future, use the generic script for something important, they would not hesitate to change the interface and break whatever I have been doing so far.
• Rafael, you are correct, the TWiki object is created in TWiki::UI.
• For "lightweight" I indeed have both alternatives in mind:
  1. Use TWiki initialisation, but no plugins: The profile binary of the BenchmarkContrib (only in SVN) does not support any plugins. It needs, however, some basic TWiki support for its access control, and its current implementation calls too much TWiki interfaces where probably TWiki::Func interfaces would do. It has a challenge which could perhaps be solved using the endPoint parameter of rest, but I haven't really investigated yet.
  2. Don't even initialize a TWiki environment: A small script pulls in data from a directory to return a XHTML snippet. This is called by an AHAH application which expands users' contact data on click, but unlike a popup uses some space of the main window. This is admittedly a border case since the script could go anywhere outside the TWiki tree as well. But it is intended to be used from within TWiki, and it doesn't return a valid HTML page, so I keep it there.
• Crawford's explanation about today's rest makes one thing evident: The REST stuff has been hacked in without an implementation of a concrete use case. So its unRESTy behaviour in case of errors was unnoticed for some time, its original intention got lost (at least to me), and Crawford simply made it fit the needs of his use case. Which are different from mine.

Let me try to summarize: Having missed the initial ideas in RESTfulPluginInterface and the current implementation of the action tracker, I thought that the intention of the rest script was to be for the bin directory what TWiki::Func is for the API: A release-independent, documented way to get some arbitrary "main program" called in a TWiki ecosystem. Under this assumption my criticism would have been justified. But given the information in the discussion above, I see that rest does what it does on purpose, and I can keep my current mechanism without much trouble.

-- HaraldJoerg - 26 Jul 2007

One thing is pretty clear; what needs refactoring is the documentation, mainly the doc of rest and twiki in TWikiScripts.

-- CrawfordCurrie - 27 Jul 2007