create new tag
, view all tags

Meta Data Driven Tag Design

TWiki is immensely powerful, and with plugins enhancing its range of capabilities, it becomes more so almost every day. There is a culture of maintaining a reasonably comprehensible and useable documentation set. However, as time goes by and the range and capabilities of plugins increases, it has become more and more difficult to remember all the options on the wide range of available tags (TWiki variables).

The SEARCH tag alone has 28 parameters. EDITTABLE has 8, and when combined with TABLE (27) that makes 35 parameters to control tables. Add to that the sophisticated table definition features, it makes it a bit of a nightmare to remember it all.

Fortunately most authors have been pretty good about documenting their contributions, and PeterThoeny recently started an initiative to create Var topics for the plugins tags that further improve the picture. However the fact remains that when editing a topic, you are constantly forced to flip back and forth to documentation. SvenDowideit did some preliminary work (the ComponentEditPlugin) on a pluggable generic tag editor in perl, for a client, but support ended and the work was never finished.

With the advent of TinyMCE there is potential for a big improvement to this picture. TinyMCE has a plugin architecture, which supports some types of feature enhancement to be plugged in to the editor. In a spirit of experiment I recently developed an action editor plugin. The plugin is illustrated in the following screenshots.

Step 1

Edit a topic containing an action, see the action highlighted in the text


Step 2

Move the cursor to the end of the text and click the running figure in the toolbar to invoke the action editor.


If the cursor is in an existing action, it is edited instead of creating a new action.

Step 3

Close the action dialog


Step 4

Click the pickaxe to flip into TML


This plugin is quite crude; it's little more than a few lines of Javascript, and only took me a saturday to write. No server side changes were required. The schema for an ACTION tag is hard-coded into the Javascript. But it illustrates the point - it is easy to develop 'tag editors' that can be plugged into TinyMCE.

Now, in order to support the wide range of tags available in TWiki and plugins, it makes sense to have a generic tag editor. The advantage of such a plugin editor would be enormous, especially from a usability perspective for recent adopters, to help ease them into TWiki application development. This isn't a new idea, it's just that the technology has moved on to a point where I think it is more achieveable.

I am looking for a sponsor for this work.


(Usual interpretations of shall, should and could)
  1. Data-driven. The solution shall be driven by a schematic specification of the tag parameters.
  2. The schema should be described using a standard syntax which can be re-used by other solutions (e.g. JSON, XML)
  3. The schema shall support representation of all standard TWiki form types (including dates)
  4. The schema shall include documentation of the paramaters
  5. The schema for a specific tag shall be made available to clients via an RPC call to the server
  6. The schema for a specific tag shall be made available to perl via an API call
  7. The schema should support definition of sub-schemas for individual paramater values e.g. table row formats
  8. The schema should allow optional formatting information to be specified
  9. The editor should support validators for parameter values
  10. The editor could support dynamic evaluation of the tag (via an RPC call) and preview of the results.
  11. The VarTAG topic for the tag should be generated from the schema (though this could be done statically by the BuildContrib)
The sophistication of validators is open to debate. I favour the idea of validators being passed as Javascript functions in the schema (which tends to suggest that JSON would be a good choice) though perhaps the ability to make RPC calls for validation is required.

Examples schema

Just to make it more comprehensible to coders, here's a JSON schema for the action tracker example:
 tag: "ACTION",
 endtag: "ENDACTION", // optional,
 description: "Action tracker action",
    parameter: "who", // name of the tag parameter
    prompt: "Who",    // prompt in the dialog
    type: "text",     // parameter type
    size: "30",       // and size, a la TWiki forms
    description: "Enter the wikiname of the person responsible",
    prompt: "Due",
    parameter: "due",
    type: "date",
    validator: function (date) { return dateInTheFuture(date); } }, // validator function
    description: "When is the action due"
    prompt: "State",
    parameter: "state",
    type: "select",
    value: "open,late,closed",
    description: "Current state of the action"
    prompt: "Description",
    body: true, // Not a parameter, value goes in the body
    type: "textarea",
    size: "12x80",
    description: "Enter a description of the action"

-- Contributors: CrawfordCurrie - 07 Dec 2007


This would be immensely useful. This would bring TWiki into the realm of Yahoo Pipes and the IBM QED demo.

-- ArthurClemens - 07 Dec 2007

... and make AnnotationPlugin trivial.

-- PankajPant - 07 Dec 2007

As Crawford and I were talking about this last week smile I'd heartily support this. I may even be able to sponsor it; we should talk about that when we get to the point where we can come up with some sort of estimate for the work involved to build the framework and some of the more common tag editors.

I guess a sensible approach would be to target tag editors for the variables and plugins that ship with TWiki by default; ultimately plugin authors should be responsible for setting up the meta-data for their own plugin.

For the validator functions: we should also probably look at a way of sharing common validators between different tag editors - for example, the validator in Crawford's example code above would probably be useful in a number of other tags too.

-- JasonWickham - 07 Dec 2007

Validators are a tricky problem. I had envisaged a common library of validator functions - dateInTheFuture being an example - but I haven't thought through all the implications of that. Probably the best approach would be to model it on perl's require pragma, so that validator object libraries can be imported on demand (TinyMCE has a sort of simple require equivalent already).

Even trickier is the fact that validators may be required in environments other than Javascript. For example, the spec might be used within a perl program, where a Javascript validator is useless. Perhaps we can kill two birds with one stone; the expression

    validator: "StandardValidators.dateInTheFuture",
could be used to call the dateInTheFuture method from the StandardValidators library, in a language-independent way.

Another interesting problem is data presentation. I have described a system where fields would be listed top-to-bottom in a dialog, in much the same way as TWiki forms work. However this presentation doesn't work in all cases. Sometimes you want field prompts to be side-by-side.

An approach that takes account of this would be to encode the presentation in the field specification, by writing the field specification in HTML. The HTML would then be embedded into the dialog. HTML::Parser is available for any application that wants to reverse-engineer the field specification from the HTML. For example,

Who: <input type="text" name="who" class="parameter" />
Due:<input type="text" name="due" class="date parameter" /><input type="image" name="calendar" src="../../../../../../JSCalendarContrib/img.gif" align="middle" alt="Calendar" onclick="showCalendar('action_due','%e %B %Y'); return false;" />
State:<input type="text" name="state" class="parameter" />
Description: <textarea name="description" class="body" rows="5" cols="40"></textarea>
Of course this increases reliance on both HTML and Javascript, but it does allow more flexible formatting.

I just realised that a more sensible approach would probably be to support a cssclass parameter in the spec.

-- CrawfordCurrie - 08 Dec 2007

In ComponentEditPlugin, I defined the Tag syntax in perl, specifcally because this would enable

  1. javascript editing (the primary focus of the end user of the work),
  2. automated documentation generation, and loading that is conditional on a Plugin being enabled, not just the topics being available,
  3. automated validation, conversion between syntaxtical versions, and the ability to give advice.

The following is an example of the SEARCH definition, that is then converted to JSON for use by the ComponentEditor (I'll port it to TinyMCE some time soon). Adding code and expression validators really would not be difficult.

my %syntax = (
    SEARCH => {
        DOCUMENTATION => {
            DOCCO=>'Inline search, shows a search result embedded in a topic'},
        search => {
            DOCCO=>'Search term. Is a keyword search, literal search or regular expression search, depending on the type parameter. SearchHelp has more'},
        web => {
            DOCCO=>'Comma-separated list of webs to search. The special word all means all webs that doe not have the NOSEARCHALL variable set to on in their WebPreferences. You can specifically exclude webs from an all search using a minus sign - for example, web="all,-Secretweb".'},
        topic => {
            DOCCO=>'Limit search to topics: A topic, a topic with asterisk wildcards, or a list of topics separated by comma.'},
        excludetopic => {
            DOCCO=>'Exclude topics from search: A topic, a topic with asterisk wildcards, or a list of topics separated by comma.'},
        header => {
            DOCCO=>'Custom format results: see FormattedSearch for usage, variables & examples'},
        format => {
            DOCCO=>'Expand variables before applying a FormattedSearch on a search hit. Useful to show the expanded text, e.g. to show the result of a SpreadSheetPlugin %CALC{}% instead of the formula'},
        seperator => {
            DOCCO=>'Line separator between hits'},
        type => {
            option=> ['keyword', 'literal', 'regex'],
            DOCCO=>'Do a keyword search like soap "web service" -shampoo; a literal search like web service; or RegularExpression search like soap;web service;!shampoo'},
        scope => {
            option=> ['topic', 'text', 'all'],
            DOCCO=>'Search topic name (title); the text (body) of topic; or all (both)'},
        order => {
            DOCCO=>'Sort the results of search by the topic names, topic creation time, last modified time, last editor, or named field of TWikiForms. The sorting is done web by web; if you want to sort across webs, create a formatted table and sort it with TablePlugin\'s initsort. Note that dates are sorted most recent date last (i.e at the bottom of the table).'},
        limit => {
            DOCCO=>'Limit the number of results returned. This is done after sorting if order is specified'},
        date => {
            DOCCO=>'limits the results to those pages with latest edit time in the given TimeInterval.'},
        reverse => {
            DOCCO=>'Reverse the direction of the search'},
        casesensitive => {
            type=>'onoff', default=>'off',
            DOCCO=>'Case sensitive search'},
        bookview => {
            type=>'onoff', default=>'off',
            DOCCO=>'show complete topic text'},
        nosummary => {
            type=>'onoff', default=>'off',
            DOCCO=>'Show topic title only'},
        nosearch => {
            type=>'onoff', default=>'off',
            DOCCO=>'Suppress search string'},
        noheader => {
            type=>'onoff', default=>'off',
            DOCCO=>'Suppress search header '},
        nototal => {
            type=>'onoff', default=>'off',
            DOCCO=>'Do not show number of topics found'},
        zeroresults => {
            type=>'onoff', default=>'off',
            DOCCO=>'Suppress all output if there are no hits'},
        noempty => {
            type=>'onoff', default=>'off',
            DOCCO=>'Suppress results for webs that have no hits.'},
        expandvariables => {
            type=>'onoff', default=>'off',
            DOCCO=>'Expand variables before applying a FormattedSearch on a search hit. Useful to show the expanded text, e.g. to show the result of a SpreadSheetPlugin %CALC{}% instead of the formula'},
        multiple => {
            type=>'onoff', default=>'off',
            DOCCO=>'Multiple hits per topic. Each hit can be formatted. The last token is used in case of a regular expression ";" and search'},
        nofinalnewline => {
            type=>'onoff', default=>'off',
            DOCCO=>'If on, the search variable does not end in a line by itself. Any text continuing immediately after the search variable on the same line will be rendered as part of the table generated by the search, if appropriate.'},
        recurse => {
            type=>'onoff', default=>'on',
            DOCCO=>'Recurse into subwebs, if subwebs are enabled.'},

-- SvenDowideit - 08 Dec 2007

I took the liberty of reformatting your example to be a bit more readable.

There were a number of (unwritten, sorry) reasons I used JSON rather than perl for the description.

  1. the realisation that embedding a syntax description into a plugin isn't going to work, as tags are implemented in several places in code (TWiki.pm, various other code modules, plugins and contribs). Since the syntax description can't be embedded, the requirement to use perl isn't a strong one. JSON is a much more readable syntax than perl, IMHO, as you don't clatter into problems like variable interpolation in strings.
  2. I want to serve this content to the Javascript as fast as possible, and had it in mind to do that directly from the pub area, thus bypassing the requirement to invoke a REST script.
  3. I want to be able to process tag specs without loading the code that implements them - for example, during a build.pl.
There needs to be a way to abstract the tag definition out of code. I had in mind something similar to the Config.spec file; for example, we could adopt a standard of pub/tags containing a set of tag definition files, each named TAGNAME.js.

On the flip side,

  1. using JSON to represent tag specs forces the perl code to load CPAN:JSON to read the specs (while it's trivial to read the JSON spec without using the JSON module, the principle of reuse militates against that)
  2. not using a REST handler to serve the spec means that TWiki variable expansion isn't possible in the spec, which could make defaults very clumsy

-- CrawfordCurrie - 10 Dec 2007

I think your reasons are not particularly strong, compared to:

  1. TWiki is written in Perl, so Perl structures are more re-usable than JSON
  2. TWiki can convert those definitions to whatever form is needed later (I'd use configure to do it)
  3. The definitions could use non-dynamic TML that is then converted.
  4. the definitions could be loaded dynamically as per the enabling / disabling of plugins
Mind you, I like the idea of a TWIKITAG.spec. to go fully lightweight, we could even have that spec file optionally contain both perl and js code to render / edit the tag, not just the docco and parameter validation info. Then, when the tag is enabled in configure, all the perl code for all the lightweight tags would be concatinated together into one compressed perl module, the VarTWIKITAG topic created, and any relevant json/tmpl or whatever files be generated.

bloomin neat.

-- SvenDowideit - 11 Dec 2007

OK, I can go with that. "When the tag is enabled in configure" is treading into the murky waters of Meredith's tag code. At the moment I'm assuming that the only server-side code required will be a REST handler that delivers the spec (and that will be in a contrib so I can be compatible with 4.2). I might do generation of the VarTAG topic, but statically in BuildContrib, in the interests of keeping it simple. Damn, I wish we had a way of initialising contribs!

-- CrawfordCurrie - 11 Dec 2007

I have done a fairly detailed analysis of how long it would take me to implement this:

Design 4h
REST handlers and other server-side bits 12h
Generate .tag specs for ~50 core tags, ~10 mins each 8h
JS framework for dialog generation 10h
Testing, polishing 16h
Total 50h

-- CrawfordCurrie - 12 Dec 2007

I would say: put a project donate button up. Not just a bag with a hole, but one with a specific goal.

-- ArthurClemens - 12 Dec 2007

That's not a bad idea. Watch out for PledgePlugin!

-- CrawfordCurrie - 13 Dec 2007

50h work huh? That's about 2 days then, right Crawford? wink Let me see what I can do...

-- JasonWickham - 14 Dec 2007

Edit | Attach | Watch | Print version | History: r14 < r13 < r12 < r11 < r10 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r14 - 2008-02-17 - SvenDowideit
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.