In my view, this is a bug - expanding such variables doesn't really qualify as 'verbatim', and it makes it hard to embed templates or TWiki code in TWiki pages... See TWikiDrawApplet for an example where the included patch is messed up by this.

The same comments also apply to other VERBATIM text processing, e.g. in Preview mode anything looking like a URL within VERBATIM is expanded.

Ideally this fix would apply to Preview mode as well, so the user can accurately preview what the saved page will look like.

Has anyone got any comments on why this shouldn't be changed? I suspect nobody is actually relying on the current behaviour, instead they are interspersing <NOP> elements or dots, as in TWikiTemplatingSystem, which wouldn't be changed by this fix.

-- RichardDonkin - 22 Jul 2001

Please can we get this fixed for the new TWiki release? It is fairly annoying, and it's hard to explain to users why this happens.

-- RichardDonkin - 25 Aug 2001

See also:

• VerbatimModeForSourceCodes

No, it isnot possible to escape variable with the <verbatim> tag because of timing. Variables are evaluated first, then the rendering happens in a different function. You can escape variables with <nop> if needed, i.e. %<nop>EDITBOXHEIGHT%

-- PeterThoeny - 04 Feb 2001

Variables are now expanded by the s/// command, doing it's job very quickly. Having to check if it's in verbatim mode will cause the system to slow..

However, IMHO there should be a way to

• quickly expand the variables with s///; and
• have the variables not expand within verbatim mode

The current proces is (roughly):

• expand variables
• parse/render resulting text

What I'm thinking of is

• parse text for verbatim mode text
  • replace variables in this mode by a non expanding variable
    eg. %TOPIC% becomes %#TOPIC=
• expand variables (no changes here)
• parse/render resulting text
• revert verbatim variables back to normal using s///
  eg. %#TOPIC% becomes =%TOPIC% again
  • NOTE: all %#variable% variables will be set to %variable%

-- HansDonner - 26 Aug 2001

I've did some modifactions in TWiki.pm. Modifications marked bij # HD - VerbatimExpandsVariables. For now it seems to work, but I have tested it only very roughly.

sub handleCommonTags
{
    my( $text, $theTopic, $theWeb, @theProcessedTopics ) = @_;

# HD - VerbatimExpandsVariables - Begin
    # derived from getRenderedVersion
    my( $result, $insideVERBATIM );

    $result = "";
    $insideVERBATIM = 0;
 
    if( m|< verbatim>|i ) {
        # no point wasting time if there's no verbatim 
        foreach( split( /\n/, $text ) ) {
            # change state:
            if( m|< verbatim>|i ) {
              $insideVERBATIM = 1;
            }
            if( m|</ verbatim>|i ) {
              $insideVERBATIM = 0;
            }
            if( $insideVERBATIM ) {
                # Replace %VARIABLE{"parameter"}% with %#VARIABLE{"parameter"}%
                s/%(.+%)/%#$1/go;
            }
            $result .= $_;
        }
        
        $text = $result;
     }
# HD - VerbatimExpandsVariables - End

    # PTh 22 Jul 2000: added $theWeb for correct handling of %INCLUDE%, %SEARCH%
    if( !$theWeb ) {
        $theWeb = $webName;
    }

    # handle all preferences and internal tags (for speed: call by reference)
    $includingWebName = $theWeb;
    $includingTopicName = $theTopic;
    &TWiki::Prefs::handlePreferencesTags( $text );
    handleInternalTags( $text, $theTopic, $theWeb );

    # recursively process multiple embeded %#INCLUDE% statements and prefs
    $text =~ s/%#INCLUDE{(.*?)}%/&handleIncludeFile($1, $theTopic, $theWeb, @theProcessedTopics )/geo;

    # Wiki Plugin Hook
    &TWiki::Plugins::commonTagsHandler( $text, $theTopic, $theWeb );

    # handle tags again because of plugin hook
    &TWiki::Prefs::handlePreferencesTags( $text );
    handleInternalTags( $text, $theTopic, $theWeb );

    $text =~ s/%#TOC{([^}]*)}%/&handleToc($text,$theTopic,$theWeb,$1)/geo;
    $text =~ s/%#TOC%/&handleToc($text,$theTopic,$theWeb,"")/geo;

# HD - VerbatimExpandsVariables - Begin
    # restore %#VARIABLE{"parameter"}% back to %VARIABLE{"parameter"}%
    $text =~ s/%#(.+%)/%$1/go;
# HD - VerbatimExpandsVariables - End

    return $text;
}

Bugger, when adding this code, I noticed that the verbatim mode has also problems with it's own tags....

See also

• NestedVerbatimTags

-- HansDonner - 26 Aug 2001

I've done a slight variant of the above that doesn't process text in vertabtim tags. Simply subsitute the following for the text sub handleCommonTagsCore. Worth putting in core (before/after current release)? Note I've dealt with nesting here, although getRenderedVersion still doesn't do this.

# =========================
sub handleCommonTags
{
    my( $intext, $theTopic, $theWeb, @theProcessedTopics ) = @_;
    
    if( $intext !~ /< verbatim>/i ) {
        return handleCommonTagsCore( $intext, $theTopic, $theWeb, @theProcessedTopics );
    }
    
    # Exclude text inside verbatim from variable substitution
    
    my $tmp = "";
    my $outtext = "";
    my $verCount = 0;
    
    foreach( split( /\n/, $intext ) ) {
        if( /< verbatim>/i ) {
            $verCount++;
            if( $verCount == 1 ) {
                $tmp = "";
                $outtext .= handleCommonTagsCore( $tmp, $theTopic, $theWeb, @theProcessedTopics );
            }
        } elsif( m|</ verbatim>|i ) {
           $verCount--;
        }
        if( $verCount ) {
            $outtext .= "$_\n";
        } else {
            $tmp .= "$_\n";
        }
    }
    
    $outtext .= handleCommonTagsCore( $tmp, $theTopic, $theWeb, @theProcessedTopics );

    return $outtext;
}

# =========================
sub handleCommonTagsCore

-- JohnTalintyre - 28 Aug 2001

• [ HansDonner ] Added a space in the verbatim tags in order to see them here.... Remember to remove them when pasting this into your code

There was a mistake in the above fix that stopped %TOC% working correctly. Here's code that works. Rename handleCommonTags to handleCommonTagsCore and insert the follow sub. Remove spaces in "< verbatim" amd "< /verbatim> before using.

# =========================
sub handleCommonTags
{
    my( $intext, $theTopic, $theWeb, @theProcessedTopics ) = @_;
    
    if( $intext !~ /< verbatim>/i ) {
        return handleCommonTagsCore( $intext, $theTopic, $theWeb, @theProcessedTopics );
    }
    
    # Exclude text inside verbatim from variable substitution
    
    my $tmp = "";
    my $outtext = "";
    my $nesting = 0;
    my $verbatimCount = 0;
    my @verbatim = ();
    
    foreach( split( /\n/, $intext ) ) {
        if( /< verbatim>/i ) {
            $nesting++;
            if( $nesting == 1 ) {
                $outtext .= "%_VERBATIM$verbatimCount%\n";
                $tmp = "";
                next;
            }
        } elsif( m|</ verbatim>|i ) {
            $nesting--;
            if( ! $nesting ) {
                $verbatim[$verbatimCount++] = $tmp;
                next;
            }
        }

        if( $nesting ) {
            $tmp .= "$_\n";
        } else {
            $outtext .= "$_\n";
        }
    }
    
    $outtext = handleCommonTagsCore( $outtext, $theTopic, $theWeb, @theProcessedTopics );
    
    for( my $i=0; $i<$verbatimCount; $i++ ) {
        my $val = $verbatim[$i];
        $outtext =~ s|%_VERBATIM$i%|< verbatim>$verbatim[$i]</ verbatim>|;
    }
    
    return $outtext;
}

Any objections to adding this bug fix to the alpha (CVS)?

-- JohnTalintyre - 10 Nov 2001

I hope there aren't! I look forward to seeing this fixed.

-- RichardDonkin - 13 Nov 2001

I'm ready to put this into CVS, but have would like thoughts on how it will affect getRenderedVersion. At present I've gone for using the same routine as above and I've done it so Plugins never see the text within the <verbatim> tags. So:

• there's something of an efficiency issue as check is done for vebartim twice as full check on topic text - difficult to fully deal with this easily as variable processing and rendering are quite distinct. I could try and refactor code so processing is more efficient during rendering, but it would make things a bit more complicated.
• should plugins get a chance to altered the verbatim text?

-- JohnTalintyre - 03 Jan 2002

Great to see this going into CVS! I'm not sure how big the efficiency issue really is, but IMO correctness is more important - most of the overhead on TWiki requests seems to happen due to CGI launching a fresh Perl interpreter, so I suspect the extra delay due to 2 scans of the topic is not too bad.

As for plugins, I think that verbatim text is one of the few cases where no plugin should be able to change the text (by definition really). Someone can always define a new verbatim-like tag using a different syntax if they want this.

-- RichardDonkin - 04 Jan 2002

Your reason looks fine to me Richard, although I always run under ModPerl and hence don't have interpreter launch overhead.

-- JohnTalintyre - 04 Jan 2002

Fix commited to CVS.

-- JohnTalintyre - 09 Jan 2002

Good stuff

-- RichardDonkin - 09 Jan 2002

Bugger, I wanted it to expand % variables. I have my website's global javascript file as a topic and I use the verbatim tags, but I also need an include to work, so I can put external data in the javascript source...

-- AndrewTetlaw - 19 Jan 2002

Andrew, can you give a bit more information about how you use verbatim. Why do you want an INCLUDE within a verbatim block to work?

-- JohnTalintyre - 19 Jan 2002

Hi, John. I made a topic about it: UsingTopicToDefineJavaScriptSourceFile. I need to be able to make a topic and then use it's contents verbatim, that is with no LinkWords,

tags and so on. Uses such as global CSS file (UsingTopicToDefineCSS) or JavaScript source file are the most immediate needs. I want to be able to reference the topic by a URL or %INCLUDE%. Perhaps I'm asking to have my cake and eat it too when I say I don't want any automatic rendering of topic names, http links, line spacing and so on but I DO want to use % variables. So I've used verbatim tags to avoid automatic rendering.

Specifically I have a global javascript in a topic and I use //<verbatim> so it doesn't get links and line breaks rendered and the tag is commented out in the JavaScript. In the main template I use an %INCLUDE to pull in this topic between <script> tags (could have also referenced it via URl for an external JavaScript source file). But in the script I have an array I want populated by values from another topic. Mainly so that I can allow users to modify the list of values that go into the array, in a seperate topic, without getting their hands on the javascript topic. I attempted to put array = [%INCLUDE("topic")%]; in the JavaScript but of course, it came out verbatim! So I guess the above arguments are logically sound. Verbatim does exactly what it says. Just doesn't help me is all

-- AndrewTetlaw - 20 Jan 2002

Andrew - how about having verbatim tags in most of the JavaScript, but leaving them out where you want variable expansion? The TWiki variables could be kept in one small section that just initialises JavaScript variables.

PeterThoeny pointed out that the recent code change didn't work for included topics. Now fixed and redundant code in getRenderedVersion removed. Note that this code alters the number of line in the text, as it strips away the verbatim sections when processing. Does this cause a problem for some of your plugins? I would hope they need to be immune to this as variable expansion and other text processing can also affect the number of lines.

-- JohnTalintyre - 23 Jan 2002

Thanks, John. That works fine. So simple, I feel like a dunce! Have updated UsingTopicToDefineJavaScriptSourceFile.

I don't know of any plugins we use that use the line count.

-- AndrewTetlaw - 24 Jan 2002

Re-opening this one... The following use of Verbatim does expand the variable (in Preview page only) for some reason - the saved page is OK. You'll need to do Edit and Preview on this page to see the bug.

?t=% GMTIME{\"\$mo\$day\$hours\$minutes\$seconds\"}%

... turns into ...

?t=%GMTIME{\"\$mo\$day\$hours\$minutes\$seconds\"}%

-- RichardDonkin - 22 Mar 2002

Now should be fixed in alpha, see SubstitutionInVerbatimPreview

-- JohnTalintyre - 25 May 2002

I just found a related bug: See NopExpandsVariables for details.

-- TomKagan - 27 Nov 2002