Refactoring handleCommonTags and handleIncludeFile

These two closely-related functions are rather confusing, and there are some performance concerns with their behavior as well. For example, handleInternalTags is called a large number of times in every request, especially where includes are present. handleInternalTags is a long function performing a large number of substitutions on the entire topic text; depending on the topic length, number of includes, and how Perl optimizes the function, the processing time expended just in this function could become quite large.

Also, the degree of processing isn't very orthogonal: included text will get processed more often by handleInternalTags and commonTagsHandler than other text. If the aim is to support recursive evaluation of %-expressions, this goal is satisfied only to a degree; depending on the location of the expression it will be re-evaluated different numbers of times. Proper implementation of this would require some way to tell whether a variable substitution has occurred (and thus further processing is required). While this might be possible for TWiki-internal expressions, the plugin API has no way of providing this information. Even changing the plugin API would not help: current plugins would still be supported, and the only way to deal with them logically is just to set a certain maximum level of recursion and stick to it.

If recursive evaluation is not the goal, then the process can be made much more efficient by changing the order of certain calls and reducing the amount of processing done in handleIncludeFile; ideally, handleIncludeFile would only process nested includes, and would be called before any variable processing. Then variable and plugin commonTagsHandler processing would be done after all included files had been read in. Or, alternately, handleIncludeFile would be invoked last, and would simply recurse back to commonTagsHandler to handle nested includes and further variable substitution within the included text. Either way, the result would be much more elegant than the current setup.

-- WalterMundt - 31 Jan 2004

After reading PreviewExpandsVariablesTwice, I'm inclined to believe that there are some similar bugs with includes due to my comments above...I'll check it out on my local sandbox and see.

Indeed, on a normal view regular preferences are already expanded twice, and for every level of include they go through another expansion after those two. Look at PreferencesTest1 vs. PreferencesTest2 to see what I mean.

-- WalterMundt - 10 Mar 2004

Changes in this are need to be done very carefully since it might break existing TWikiApplication that make assumptions on variable expansions.

-- PeterThoeny - 11 Mar 2004