Not All Tags Work With %INCLUDE%

When including a section or topic in another topic, the included text is not processed in the same context as when rendered on its original page. This may lead to some surprising consequences in plugins, depening on which handler is invoked.

Examine the results of the following experiment: I created a plugin which just prints out the values of various tags related to the current topic. There is a plugin tag for each of the rendering related handlers, as shown in the table below. Each plugin tag takes the following parameter: topic, including, and base, and I pass %TOPIC%, %INCLUDINGTOPIC%, and %BASETOPIC%. Then it prints out the values for the passed variables, plus the value of $topic, $session->{SESSION_TAGS}{INCLUDINGTOPIC}, $session->{topicname}, and $session->{SESSION_TAGS}{TOPIC}.

In other words, the plugin tag is invoked, for example, with

%TESTCOMMONTAGSINCLUDE{ topic="%TOPIC%", including="%INCLUDINGTOPIC%, base="%BASETOPIC%" }%

Here are the results, assuming the name of the included topic is IncludedTopic and the name of the including topic is BaseTopic

This can really bite plugin developers and does not appear to be quite to spec....

The explanation of above is due to that the included text is inserted during the commonTagsHandler. So after that, the fact that text was included is lost.

The problem is that there are plugins (e.g., MultiEditPlugin, SectionalEditPlugin, EditTablePlugin, EditTablerowPlugin, etc.) which need to know what topic the text being edit really is in, as they insert code that has to go to that topic, not the including topic (e.g., to edit that topic). That is no problem for EditTablePlugin and EditTablerowPlugin, as these plugins make these text manipulations in commonTagsHandler. However, the other plugins mentioned do their heavy lifting in preRenderingHandler, and here the topic information is lost.

One way I can see to deal with this is to insert when in commonTagsHandler the topic information literally into the text of the manipulated topic, so that it can be detected later. I'd prefer a way of storing that information in a variable but cannot quite see how this would work when more than one topic is included. Take a look at MultiEditPlugin how silly a work around I had to implement.

Note further that preRenderingHandler does insert an extra newline in the output, and that postRenderingHandler does not render the included text, which I guess should be expected (but might still come as a surprise). In particular, note that the included text is not handled at all by the beforeCommonTagsHandler.

There are three issues here:

• A plugin author does not have access to the full topic before portions of it are included. Therefore, if information from non-included portions are required to correctly render the included portion, the plugin will fail.
• A plugin author (and the core) does not have an understanding where the included text begins and ends. Therefore, text manipulation that should apply only to one or the other will fail, as the boundaries are lost.
• The impact of inclusion on what is available in handlers and what tags mean is not explained in the documentation for plugin writers nor in the documentation of the tags, where appropriate.

-- Contributors: ThomasWeigert - 11 Nov 2006

Discussion

%INCLUDE is a macro instruction, like #include in the c pre-processor (cpp). %INCLUDE says "include the text of this other thing verbatim in this topic and keep on processing". There is some jiggery-pokery to expand certain tags in the included topic differently (such as %BASETOPIC), but otherwise the boundaries between the including text and the included text is deliberately blurred. As you say, the fact that the text was included is lost. This is exactly why cpp is able to inject # file line number type information into the processed source. The lexer is smart enough to remove it again when it parses.

However, this is programmer thinking. Most plugin authors do not want to contend with embedded synch statements in the source. Further, it is my impression that plugins needing this sort of information are quite rare. The core already provides the commonTagsHandler and the preRenderingHandler to support processing at the different stages of the pipeline. It seems to me that the problem here is in the plugins, not the core. A plugin requiring instrumented source should be able to perform the instrumentation during the commonTagsHandler, and remove the instruments again in the preRenderingHandler.

Of course it would help to have

1. a standard for instrumentation, so that plugins can share the instruments (perhaps a InstrumentationContrib is in order here) and also
2. the ability to stack tag handlers (so a plugin can registerTagHandler an extra layer of processing over the default %INCLUDE).

As a final point, here be dragons. Remember that a %INCLUDE can generate new tags e.g. I could say %SEARCH{%INCLUDE{"SearchParams"}%}% such that in different webs SearchParams contains different search parameters. If you instrument this (e.g. by embedding # file statements, you are likely to break the generated SEARCH.

-- CrawfordCurrie - 12 Nov 2006

Crawford, your analysis is right but my concern is not instrumentation but manipulation of the included text.

The problem is that the plugins cannot infer needed information, as they do not have a chance to examine the whole completed topic. Please see NeedBeforeIncludeHandler for further explanations. But the long and short is that the plugins only get the included text (in commonTagsHandler) after it has been excerpted from the possibly much longer topic that is being included. At this point, critical information for the plugins to do their work may not be accessible any longer in the included text. My diagnosis at NeedBeforeIncludeHandler is that we need a plugin handler that allows access to the complete topic being included before TWiki::_INCLUDE strips it down to the actually included text. Either a new handler or applying the beforeCommonTagsHandler (which is currently not applied to included text) would do.

I have implemented sectional editing of included files with the help of such new handler.

-- ThomasWeigert - 12 Nov 2006

The second part of this note pertains to that none of the above is documented, but plugin authors (e.g., myself) may not be aware of the impact of inclusion. I bet you that there are a number of plugins out there which work great but will break on included text because of the dependency of the meaning of certain tags on where in the rendering pipeline we are and also the fact that beforeCommonTagsHandler is mysteriously not applied to included text.

-- ThomasWeigert - 12 Nov 2006

OK, that's different. I would personally prefer that you didn't have to add another handler, but instead overrode _INCLUDE, but I can see why adding a handler is a more appealing approach.

You would need to refactor _INCLUDE so that the handler can be called after the included context is established, but before the sections are processed. Note that the current ordering of tag handling in _INCLUDE is quite delicate; I had a frightful time maintaining compatibility.

BTW the reason beforeCommonTagsHandler is not applied to included text is simply because I didn't apply it. But if I had, it would have been applied after STARTSECTION processing. STARTSECTION is not a "common tag" by any definition.

-- CrawfordCurrie - 13 Nov 2006

Well, obviously nobody missed it not being applied. We can have this handler do something usefull if we apply it to the included topic before inclusion. There really is no right or wrong here, I think.

What is the definition of "common tag"?

-- ThomasWeigert - 13 Nov 2006

Gosh, that's a tough one. "Any tag that stands on it's own (doesn't require a matching close tag) and isn't an unregistered plugin tag" is the best I can do. "registered tag" would really be a better name, but the terminology was established pre-Athens.

-- CrawfordCurrie - 08 Jan 2007