Feature Proposal: Set and Get Variables

Motivation

Offer a powerful way to set and get variables with the goal to strengthen TWiki as an application platform. The SpreadSheetPlugin already has $SET() and $GET() to manipulate variables. It has some limitations though: No complex structures (arrays, hashes etc), no newlines in variables, and no control over whitespace.

Consider the set/get variable feature a first step towards a powerful template language like TemplateToolkit or JSP with objects, methods, blocks, loops, filters and more.

Description and Documentation

Add 2 new TWiki variables to set and get variables, with scope of current topic (which might include other topics):

• %SET{ "example" value="....." }% -- set a variable
• %GET{ "example" }% -- get a variable

Variables can also stored persistently and retrieved later in any topic using this syntax:

• %SET{ "example2" remember="1" }% -- save a variable persistently for later retrieval
• %GET{ "example2" }% -- load a saved variable

-- Contributors: PeterThoeny - 2011-03-27

Future enhancement ideas

As for syntax, I first considered native Perl variable syntax for easy of implementation, such as:

%SET{
    "example"
     value="{
        title => 'Wikis for Dummies',
        authors => [
            { id => 'dan',   name => 'Dan Woods',    city => 'New York' },
            { id => 'peter', name => 'Peter Thoeny', city => 'San Jose' },
        ],
        isbn10 => 0470043997,
        primes => [ 2, 3, 5, 7, 11, 13 ],
    }"
}%

Not everybody knows Perl, and it is not a web standard for I/O. JASON is pervasively used and well understood. It lends itself to get and set variables in TWiki. For example a SEARCH can produce a search result in JSON format that can be stored away and consumed later on by a browser-side JavaScript program.

%SET{
    "example": {
        "title": => "Wikis for Dummies",
        "authors": => [
            { "id": "dan",   "name": "Dan Woods",    "city": "New York" },
            { "id": "peter", "name": "Peter Thoeny", "city": "San Jose" },
        ],
        "isbn10": => 0470043997,
        "primes": => [ 2, 3, 5, 7, 11, 13 ],
    }
}%

Above example is pure JSON syntax. Unfortunately it is not compatible with the standard TWiki variable parameter syntax. If we use the TWiki standard it would look like this:

%SET{
    "example"
    value='{
        "title": => "Wikis for Dummies",
        "authors": => [
            { "id": "dan",   "name": "Dan Woods" },
            { "id": "peter", "name": "Peter Thoeny" },
        ],
        "isbn10": => 0470043997,
        "primes": => [ 2, 3, 5, 7, 11, 13 ],
    }'
}%

%SET{
    "example"
    value="{
        \"title": => \"Wikis for Dummies\",
        \"authors": => [
            { \"id\": \"dan\",   \"name\": \"Dan Woods\"    },
            { \"id\": \"peter\", \"name\": \"Peter Thoeny\" },
        ],
        \"isbn10\": => 0470043997,
        \"primes\": => [ 2, 3, 5, 7, 11, 13 ],
    }"
}%

This syntax is not so intuitive for people who know JSON. It is also not very robust, as shown in above example. For example if you use single quotes for value parameter you need to escape single quotes within the value. If you use double quotes you need to escape all double quotes in the value, which is not practical.

Example

This TML:

%SET{ weekdays = [ 'Mon', 'Tue', 'Wed', 'Thu', 'Fri' ] }%
Weekdays array (raw output):
%GET{ weekdays }%
-----
%SET{ "users": {
     "tom":   "Thomas",
     "dick":  "Richard",
     "larry": "Lawrence",
   }
}%
Users hash (raw output):
%GET{ users }%
-----
Bullet list output:
%FOREACH{ u IN users }%
   * %GET{ u.key }% : %GET{ u.value }%
%ENDFOREACH%

would produce the following output:

Note: The FOREACH and ENDFOREACH variables are for illustration purposes only, they are out of scope if this proposal.

Note on implementation: CPAN:JSON can be used to convert to/from Perl/JSON. It can be shipped with TWiki as a Pure Perl module (JSON::PP). Compiled C based CPAN:JSON::XS can be installed for better performance.

-- Contributors: PeterThoeny - 2010-06-18

Discussion

This could be implemented in the core or as a plugin. If as a plugin and if only set/get variables are considered we could name it SetGetVariablesPlugin, if we consider a future template engine for TWiki we could name it TWikiTemplatePlugin.

Thoughts?

-- PeterThoeny - 2010-06-19

As for storing the variables persistently, there are two options:

• Store as meta data in topic: Is version controlled (might be "noisy" with change notifications), easy to implement, no variable loss issue due to topic rename, obeys access control.
• Store in work area in a Web/Topic/Variable tree structure: opposite of first option.

-- PeterThoeny - 2010-06-19

One important point to consider: Storing variables persistently in the work area performs better than storing them as topic meta data.

-- PeterThoeny - 2010-06-19

Instead of SAVEVAR/LOADVAR we could add a persistent="on" parameter to the SET variable. Once a variable on topic scope has been declared as persistent it remembers the value between page invocation.

Performance needs to be considered: Checking if variable is persistent or not on each GET is expensive.

-- PeterThoeny - 2010-06-19

Another idea: Store values persistently as topic preferences.

-- PeterThoeny - 2010-06-22

This is now accepted by 7-day feedback period.

Allthough accepted, the spec for persistently storing and retrieving needs to be fleshed out.

-- PeterThoeny - 2010-06-28

I will rename the plugin from TWikiTemplatePlugin to TWikiServerPagesPlugin, aka TSP plugin. This with the (ambitious?) end-goal to bring the TWiki application platform on par with JSP and ASP.

-- PeterThoeny - 2010-07-02

Refocusing scope: First implementation is now SetGetPlugin that has just the SET and GET functionality, handled by TWikibug:Item6597.

-- PeterThoeny - 2010-10-17

Idea for enhancement: Add REST handlers so that jQuery can interact with persistent variables at runtime!

-- PeterThoeny - 2011-02-14

The SetGetPlugin can now optionally store variables persistently in the working area. I updated the spec above to reflect the syntax of the plugin.

I will add this plugin to the core distribution since it is an accepted proposal.

-- PeterThoeny - 2011-03-27

I split out AddSetGetPluginToCoreDistribution as a separate proposal so that plugins added to the core TWiki distribution can be tracked more easily.

-- PeterThoeny - 2011-03-27