Feature Proposal: Add GluePlugin functionality to the Core

Motivation

Make TWikiML readable: SEARCH expressions, as well as other tags really get messy up to unmaintainable when everything has to be writen in one line. This is at least uggly.

Description

I've written a mini-mini plugin, the GluePlugin, that really makes writing TWikiML a lot easier. Actually this plugin is so small and so usefull that it is only consequent to think about adding it to the core rendering engine.

-- MichaelDaum - 15 Aug 2005

Impact and Available Solutions

Documentation

If necessary, user documentation of new features introduced by this proposal.

Examples

An arbitrary real-live example from my blogging setup

%STARTINCLUDE%
~~~ %SEARCH{"^%NAME%$" type="regex" scope="topic" nonoise="on"
~~~     format="%INCLUDE{"RenderBlogEntryFormat" 
~~~                       NAME="$topic" 
~~~                       AUTHOR="$formfield(BlogAuthor)"
~~~                       HEADLINE="$formfield(Headline)"
~~~                       CATEGORY="$formfield(SubjectCategory)"
~~~                       DATE="$formfield(Day)
~~~                             $formfield(Month)
~~~                             $formfield(Year) -
~~~                             $formfield(Time)"
~~~                       TEASER="$formfield(Teaser)"
~~~                       TEXT="$formfield(Text)"
~~~                      }%"
~~~ }%
~~~ %STOPINCLUDE%

Implementation

Any comments on how the feature is implemented or could be improved

---

Discussion:

Hmmm. With the GluePlugin that makes three "standards" for continuation lines.

   * bullets can be continued using
   \n   text
   * backslashes at the ends of lines \
can be used everywhere, I think
   * and now GluePlugin \
---- adds this, (which conflicts with horizontal bar)

• bullets can be continued using \n text
• backslashes at the ends of lines can be used everywhere, I think
• and now GluePlugin ---- adds this, (which conflicts with horizontal bar)

I would prefer we standardised on just one - the backslash at the end of line. We have to support the bullet form for legacy. But please lets not add a third form!

-- CrawfordCurrie - 16 Aug 2005

Ehm, it's not three minuses (---), it is three tildes (~~~), so it does not conflict with horizontal bars. Backslashes are used for so many other things in SEARCH expressions already.

Note, that tilde glue consumes the whitespaces around it. Backslashes must not do that.

Au contrair, a backslash at the end of a line easily fails if there is more whitespace following til the real end of line. This makes lineformatting rather experimental and an unpleasant experience.

Tilde glue

1. is fun,
2. is kind of foolproof,
3. is enhancing readability and maintainability of TWikiML.

-- MichaelDaum - 16 Aug 2005

I don't like it for 2 reasons

1. its making yet another special character
2. to me ~~~~ looks like an underline, just like --- does, so the form does not suggest function.

I would be considerable more comfortable with the following syntax being legal, and if you really really needed a eat whitespace continue next line, \\ or something related to \

%STARTINCLUDE% 
 %SEARCH{"^%NAME%$" type="regex" scope="topic" nonoise="on"
     format="%INCLUDE{"RenderBlogEntryFormat" 
                       NAME="$topic" 
                       AUTHOR="$formfield(BlogAuthor)"
                       HEADLINE="$formfield(Headline)"
                       CATEGORY="$formfield(SubjectCategory)"
                       DATE="$formfield(Day)$formfield(Month)$formfield(Year) - $formfield(Time)"
                       TEASER="$formfield(Teaser)"
                       TEXT="$formfield(Text)"
                      }%"
 }%
 %STOPINCLUDE%

to be honest, our not supporting TWikiVariables over multiple lines seems like lazyness.

Also note, that your lack of space after the - in the DATE definition shows to me that ~~~ is ambiguous, as is eats 'some' whitespace, but not others?

-- SvenDowideit - 16 Aug 2005

Ambiguous? Explain. The tilde glue in the DATE work just fine.

These are the questions:

1. Does it conflict with similar means to ease writing TML, eg EOL escape?
2. Is it only curing symptoms and not the real problem?
3. How hard is it to do it right?
4. Does it break other stuff?
5. Do you detest it for aesthetic reasons?

Note, whatever TheRightThing is that it has to compare in effort to adding one regex for tilde glue s/\s*[\n\r]+~~~(.)\s*/$1/gos somewhere in handleInteralTags(). No more.

-- MichaelDaum - 16 Aug 2005

I know it probably feels like you are under attack from all sides, but please bear with us; it is really important to get it right.

1. Yes, exactly - it is quite different. \ isn't great, but it's already there.
2. No, You have accurately identified a requirement, we are just discussing what is the right solution.
3. As you know, it's a one-line fix in Render.pm. But we have to agree what "right" means.
4. It may break existing TWikiApplications.
5. Detest is too strong a word (for me anyway) but I do like consistency, and the proposal is inconsistent with existing syntax.

Maybe existing syntax is nasty, or too painful to use; but it is existing syntax and any syntax extension should be consistent with it. Consistency can come through a syntax extension such as \\ or \. I personally think the tilde sequence is too much of a departure from existing syntax - it is not obvious, and would be very surprising to an unwary user.

If backslashes are already used in SEARCH expressions, perhaps we need to reconsider use of \ to indicate a continuation line. I always preferred +, myself (++ for eating all whitespace except one, +* for eating all whitespace). I also like your approach of having the continuation character(s) at the start of the next line, rather than the end of the previous. It makes it all much easier to read. Contrived examples:

%STARTINCLUDE%
+* %SEARCH{
+*      "^%NAME%$"
++      type="regex"
++      nonoise="on"
+       format="%INCLUDE{
+*                       "RenderBlogEntryFormat"
+                        NAME="$topic"
+                        DATE="$formfield(Year) -
++                             $formfield(Time)"
+                        TEASER="$formfield(Teaser)"
+*                      }%" 
+* }%
%STOPINCLUDE%

| *Col 1*
+ | *Col 2*
+ | *Col 3* |
| Data 1
+ | Data 2 which is very long so it doesn't fit all on one line and needs
+ to be broken to remain readable for people who insist on doing their
+ wikiing on a PDA or even a mobile phone - there are some, you know, though
+ a 1024X1280 screen on a PDA would be really neat.
+ | Data 3 |

Note that we can continue to support the \ escape for as long as necessary, just by leaving the current RE in place.

-- CrawfordCurrie - 17 Aug 2005

+ instead of ~ is fine with me. But I think that +, ++ and +* have a cluttering and confusing effect. I've already missed the difference between + and ++.

Here's another try using

• +++ eat all but one whitespace
• ++* eat all whitespaces

Note, that glue aways makes up three chars. This has the advantage of your +* and is a litte less cluttering.

%STARTINCLUDE%
++* %SEARCH{
++*      "^%NAME%$"
+++      type="regex"
+++      nonoise="on"
+++      format="%INCLUDE{
++*                      "RenderBlogEntryFormat"
+++                       NAME="$topic"
+++                       DATE="$formfield(Year) -
+++                             $formfield(Time)"
+++                       TEASER="$formfield(Teaser)"
++*                      }%" 
++* }%
%STOPINCLUDE%

Further note, that most of the time you get away with using +++ in the example above rendering proper TML with one \s left in between.

++* is useful when breaking regexes on multiple lines and the like:

%SEARCH{"

++* "META:FIELD.
++*  name=\"[T]opicType\".*
++*  value=\"[^\"]*BlogEntry[,\"]\"
++* "

+++ nonoise="on" type="regex"

+++ format="   * [[$topic][$formfield(Headline)]]"

+++ }%

-- MichaelDaum - 17 Aug 2005

OK, but there are three cases: (1) eats only the newline, (2) eats all whitespace (3) eats all whitespace, spits out one space.

Can't we reduce to two characters?

1. +- eats just the newline
2. +* eats all whitespace
3. ++ eats all whitespace, spits out one space

-- CrawfordCurrie - 17 Aug 2005

Alright let's see:

+* "META:FIELD.
+*  name=\"[T]opicType\".*
+*  value=\"[^\"]*BlogEntry[,\"]\"
+* "

++ nonoise="on" type="regex"

++ format="   * [[$topic][$formfield(Headline)]]"

++ }%

And

%STARTINCLUDE%
+* %SEARCH{
+*      "^%NAME%$"
++      type="regex"
++      nonoise="on"
++      format="%INCLUDE{
+*                      "RenderBlogEntryFormat"
++                       NAME="$topic"
++                       DATE="$formfield(Year) -
++                             $formfield(Time)"
++                       TEASER="$formfield(Teaser)"
+*                      }%" 
+* }%
%STOPINCLUDE%

Hm. Making it three chars has this effect of creating a visual block that gets lost with two chars.

Anyway let's get all cases declared more precisely:

	preceding whitespaces	following whitespaces	all whitespaces
all	++-		++*
all but one			+++

• being used most of the time

Is this consistent in itself? I don't think so. And it gets harder to memorise. The "delete preceding newline" or "delete preceding whitespaces" thingy is different in a way. This is actually covered most closely by the legacy \ escape. We could add foolproofness to it to make it escape not only the next char but also all whitespaces including the EOL. It is going to stay anyway.

---

You are missing the point. \ stays for legacy, but is removed from documentation and is deprecated. It is wrong to have many syntax constructs that achieve the same thing, same as it's wrong to have inconsistent syntaxes.

Your table is wrong, BTW. ++- only deletes the immediately preceding \n, not all previous whitespace. It's also misleading. The cases are already well defined in the list:

1. +- eats just the newline
2. +* eats all whitespace
3. ++ eats all whitespace, spits out one space

And note that while you may use ++ all the time in search expressions, +- and +* are both required for correct formatting of tables, where the number of spaces is critical to the layout of the table. A user can "get by" only knowing ++, but the others are required.

-- CrawfordCurrie - 17 Aug 2005

Alright, then let's do it.

-- MichaelDaum - 17 Aug 2005

Bugs:Item260

-- CrawfordCurrie - 17 Aug 2005

I'm sorry, why don't you just want to fix the parser to be able to have multipline variables without markup?

then the only place you need this horrid syntax is inside the ="='s ???

Talking to MichaelDaum on IRC, I realised that even this is un-necessary.

%STARTINCLUDE%
%SEARCH{
      "^%NAME%$"
      type="regex"
      nonoise="on"
      format="%INCLUDE{"\
                      ""RenderBlogEntryFormat""\
                       "NAME="$topic""\
                       "DATE="$formfield(Year) -"\
                             "$formfield(Time)""\
                       "TEASER="$formfield(Teaser)""\
                      "}%" 
}%
%STOPINCLUDE%

not ideal i grant you, but as this was a 10second idea, i'm sure there is a better implementation out there.

wrt tables, it sounds like the table syntax is too brittle, and should be worked on to make it more useable (for users, and i'm pretty sure that in their case this new syntax won't help much either).

-- SvenDowideit - 17 Aug 2005

I agree with Sven. I really don't want to care if I want to eat all the whitespaces, or eat then and spit one, or eat the newline.

Would that change break old TWikiApplications?

-- RafaelAlvarez - 17 Aug 2005

We thrashed this out a bit more, and decided to fix it so TWiki Variables can be split across multiple lines, without any need for new continuation characters etc. Fixed in DevelopBranch.

-- CrawfordCurrie - 18 Aug 2005

Hmmm. This mean all plugins have to change, as just about every tag match assumes that variables are on a single line, as far as I have seen. Most plugins don't handle continuation characters either.... Now I did not check your patch to see whether it sneaks in and changes the text before you pass it to the first handler right after reading....

-- ThomasWeigert - 18 Aug 2005

No, they don't have to change. If they were to change, that would be nice, but it's not compulsory. If a plugin uses registerTagHandler they get the syntax for free.

Joining of lines ending in \ is applied after the (deprecated) startRenderingHandler and before the preRenderingHandler is called.

-- CrawfordCurrie - 18 Aug 2005

So you are saying that if a plugin does not use startRenderingHandler it will be safe?

-- ThomasWeigert - 18 Aug 2005

Crawford is mentioning 2 cases:

• If plugins tags are registered using registerTagHandler, they get multi-lines attributes for free.
• If a plugin don't register the tags, the old sintax must be used.

-- RafaelAlvarez - 18 Aug 2005

A description of the new handlers and this registerTagHandler should go into DakarReleaseNotes as well as the plugin doco.

-- ThomasWeigert - 19 Aug 2005

Ok, so the new multi-line twiki variables in DEVELOP break the GluePlugin and there's no way to fix it for DEVELOP from the perspective of a plugin ... if one still would like to have it on this engine.

Workarround for gluers on DEVELOP:

write

%
~~~SEARCH{
~~~
}%

instead of

%SEARCH{
~~~
}%

-- MichaelDaum - 19 Aug 2005

Sven proposed multi-line parameters using

%SEARCH{
   "hello" \
   "world"
}%

which would be identical to

%SEARCH{
   "helloworld"
}%

This is no extra syntax, but a reasonable use of the backslash, similar to C. Long parameters, and they do occure, are still unreadable.

-- MichaelDaum - 19 Aug 2005

Multi-line parameters are implemented, and we reckon that's good enough (at least, Michael has fallen silent )

-- CrawfordCurrie - 28 Aug 2005

Sorry, Crawford, but the multi-line feature in the current development engine is not good enough, at least not for me. Tags might span multiple lines but not their parameters. As far as I've understood you on IRC, this is not doable by changing the TML parser itself. So the GluePlugin has still its value not being obsoleted so far ;). Actually I'd say that I withdraw the proposal to integrate the _glue syntax _ into TML and let the feature remain a plugin that can be installed optionally. I've uploaded a new version of the GluePlugin that is compatible with the dakar engine now. This alters the glue syntax as proposed so far in a different way (see its docu). My personal experince with this glue thingy is ... well ... I find myself becoming addicted to it. Things are so much cleaner. TWikiApplications are much easier to write and maintain and thus might become more complex in the long run. Separating tag parameters on muliple line is a requiremet as lots of TWikiApplication programming takes place within the format parameter of SEARCH as you can see in the example above already. I've been writing "TopicFunctions" that solely expand to a format string and can be combined with other TopicFunctions returning fragments of a format string for the final SEARCH command being a TopicFunction itself. This stuff is then "called" using parametric INCLUDEs and the GluePlugin takes care that everything is recombined into a single looong line so that the TWiki engine understands it.

Here's the current set of TopicFunctions and how they look like on my experimental blogging site.

-- MichaelDaum - 28 Aug 2005

hey Michael, can you explain to me why the Glue plugin couldn't implement the

%SEARCH{
   "hello" \
   "world"
}%

syntax? (i'm curious, and a little puzzeled)

and thanks for continuing to drive this, as it is as important as you think it is!

-- SvenDowideit - 28 Aug 2005

First, any multi-line %SEARCH will be interpreted asap with no text processing plugin being able to intercept this process. The (most recent) glue plugin adds some syntax suggar to write

%~~ SEARCH{
~~~ }% 

which allows to formats stuff beyond the radar of the current TWiki parser where the tildes give you still a nice looking code block.

Second, backslashes have been used for lots of other stuff everywhere. The glue syntax is very restrictive in comparrison to that: only tildes at the beginning of a line take effect.

Third, the principle of eating whitespaces arround the tildes is simple and foolproof and allows liberal formatting of TML code by inserting arbitrary ammount of whitespace all over the place. Not so the (current) backslash implementation.

Fourth, Crawford tried to implement multi-line tag arguments and if he fails I believe that there's something serioulsy problematic with the TML parser to actually do it.

Five, ... I've forgotten.

Sixth, last not least, the GluePlugin just works and I want to go on with other stuff.

-- MichaelDaum - 28 Aug 2005

OK, no problem, I wasn't proposing ditching the plugin, I was only saying it's not going into the core for Dakar. BTW you can have newlines in parameters, wherever they are legal under the rules of TML. The difference is that the core doesn't remove such newlines, so they have to be syntactically valid.

I assume that one advantage the GluePlugin has over the core is that it can extend the syntax of plugins variables.

Oh, I didn't try to implement multi-line tag arguments; at least, not in the way presented by Sven. But as I have said several times, the syntax of "classic" TWiki tags would make it very difficult to do, I think, without a full rework of the syntax of tags. I look forward to being proved wrong!

-- CrawfordCurrie - 29 Aug 2005

I've already said this in GluePluginDev:

I proposed in SmallerSignatures using MediaWiki's syntax of using three tildes ~~~ or four tildes ~~~~ to insert just the name or name and date respectively on topic save. This would conflict with that if we ever adopt it.

-- SamHasler - 30 Aug 2005