Bug: TWiki expands RCS keywords found in text

Any RCS 'ID keywords' in the text of a page are expanded when the topic is saved/viewed.

Test case

The following line was entered as $Id$:

$Id: ExpandsRcsKeywordsInText.txt,v 1.18 2003/04/03 12:06:11 ChristianFroehler Exp nobody $

During Preview, this displays OK, but once saved, the part between the dollars is expanded by RCS.

See also TwikiOnWindowsBinaryAttachments where this happened 'in the wild'.

Environment

-- RichardDonkin - 20 Mar 2002

Follow up

The co command's -ko flag will prevent it expanding ID keywords. We could also use the -kb flag, which has same effect but also turns on binary mode on the file (perhaps useful for attachment files but we may already do that).

Workaround: Put a <nop> after the first $, e.g. $<nop>Id$. If you don't do this, the actual RCS content after the keyword name is replaced with that for the TWiki RCS file.

Fix record

I fixed this on a local Windows TWiki running Dec 2001 code, by modifying TWiki.cfg to add -ko to three lines:

$revCiCmd         = "$rcsDir/ci $rcsArg -ko -q -l -m$cmdQuote%COMMENT%$cmdQuote
-t-none -w$cmdQuote%USERNAME%$cmdQuote %FILENAME%";
#FIXME more entries need %COMMENT%
#                   RCS check in command with date :
$revCiDateCmd     = "$rcsDir/ci -l $rcsArg -ko -q -mnone -t-none -d$cmdQuote%DAT
E%$cmdQuote -w$cmdQuote%USERNAME%$cmdQuote %FILENAME%";
#                   RCS check out command :
$revCoCmd         = "$rcsDir/co $rcsArg -ko -q -p%REVISION% %FILENAME%";

This fixes the problem for both new topics and revisions of existing topics (the first two commands). The third command doesn't seem to have any effect but I put it in just in case. A bit more analysis is needed of where these commands are used to see if the third change is justified.

-- RichardDonkin - 20 Mar 2002

RcsLite won't do this, but for compatibility I'll make sure it retains the -kbo flag. Files that don't match the attachAsciiPath are stored using -kb.

-- JohnTalintyre - 20 Mar 2002

This should really be fixed... I'll test out the TWiki.cfg changes needed for the latest release.

-- RichardDonkin - 13 Nov 2002

Hmmm. I don't think this is a bug. Actually, I've been using this as feature, like so: $Date: 2003/04/03 12:06:11 $ or $Author: ChristianFroehler $ or $Revision: 1.18 $ What is the goal of eliminating this behaviour? Only RCS / CVS users would ever write these keywords, and if so, they would want to use it. And if they really, really want them verbatim, this audience should be knowledgeable enough to insert the <nop>.

2¢ PeterKlausner - 13 Nov 2002

The problem is that it actually changes the page each time someone updates it - the ID keywords at the top of this page originally looked like $Id$, then were expanded to include mine, and now yours. This is probably OK if the page is only ever updated by a single user (e.g. Wiki:PersonalWiki), in which case they can just take out the -ko options. Another example is the patch that I just included in your CacheAddOn page.

This also corrupted a version string for CGI.pm that was pasted into TwikiOnWindowsBinaryAttachments, violating the principle of 'least surprises' in my book - even though I knew TWiki was RCS-based I wasn't expecting it to modify the text of the page, losing information within a <verbatim> section.

Since you do find this useful, how about if we define a TWiki.cfg variable, $keywordExpansionFlag, that is set to -ko by default? You could set this to '' to keep the current behaviour.

An up to date set of changes to TWiki.cfg, without this extra variable, are:

    coCmd         => "$rcsDir/co $rcsArg -q -p%REVISION% -ko %FILENAME%",
    diffCmd       => "$rcsDir/rcsdiff $rcsArg -q -w -B -r%REVISION1% -r%REVISION2% -ko %FILENAME%",

-- RichardDonkin - 13 Nov 2002

Hello, For proper use of revision keywords (i.e. in a TWiki environment where it is assumed the authors really want to have them expanded), I'd think that TWiki should use an rcs "export" for page prior to editting, I'd think, which is different than binary or "don't expand" modes. This is assuming "export" is available with rcs (I know it is with cvs).

Example.

• User1 edits a page and embeds $Date$ and $Revision$, which is 1.8.
• User1 saves the page
• User2 clicks 'edit'
  • TWiki exports (instead of checks out) the page.
    • The "$Date: 11/01/2002 $" keyword text is changed to only "11/01/2002"
    • The "$Revision: 1.8 $" keyword text is changed to only "1.8"
• User2 edits the text to append or etc., perhaps adding his/her own keywords.
• User2 saves the page.
• User1 views the page.
  • The original User1 text is in-tact with the keyword data just like when he previously saved the page: "11/01/2002" and "1.8". This keyword data matches his actual edits.

In fact, the "<== This is your signature for easy copy & paste operation" could then be changed to "--Main.MyUserName - $Date$" (this text would never change) and the date would be inserted correctly by rcs on export.

-- JonathanCline - 13 Nov 2002

I'm not aware of such a feature in RCS.

-- JohnTalintyre - 14 Nov 2002

Interesting feature suggestion, but it might be quite hard to implement. Please log this as FeatureBrainstorming, since this page is a BugReport.

-- RichardDonkin - 14 Nov 2002

Since I haven't had any objections to the $keywordExpansionFlag idea raised above, I have committed this to the TWikiAlphaRelease, see CVS:lib/TWiki.cfg. Those who want to keep the current behaviour can just set this variable to '' in TWiki.cfg.

-- RichardDonkin - 15 Nov 2002

Just for the record, JonathanCline's suggestion is the -kv (value) flag to rcs's co command:

-kv    Generate only keyword values for  keyword  strings.
              For example, for the Revision keyword, generate the
              string 5.13 instead of $Revision: 1.18 $.  This can
              help  generate files in programming languages where
              it is hard to strip keyword delimiters like  $Revi-
              sion: $  from  a  string.  However, further keyword
              substitution cannot be performed once  the  keyword
              names  are  removed,  so this option should be used
              with care.  Because of this danger of  losing  key-
              words,  this option cannot be combined with -l, and
              the owner write permission of the working  file  is
              turned  off;  to  edit the file later, check it out
              again without -kv.

I am not sure it is a good idea though since it will effectively clobber the rcs id's in the file which you probably wanted to display the current data. Maybe time for a pragma plugin to allow %PRAGMA{rcskeywordexand="yes"} on a page by page basis?

-- JohnRouillard - 16 Nov 2002

I think my solution is proper. Having to set this page by page would mean it is (a) quickly forgotten, (b) quickly annoying, or (c) quickly confusing. Having this option set in TWiki.cfg just means it's (c) -- quickly confusing.

Here's my argument.

1. use of rcs keywords is a special case
   1. therefore joe average never uses it.
   2. therefore only advanced users will use it
2. advanced users may desire this feature to, for example, mark the specific revision they have edited, or the date of the edit, using "Id" or "Date" or "Revision" keywords.
3. when advanced features are used they should protect the advanced user from themselves
4. as noted above, when keywords are expanded, the topic is not "static", it changes for each edit iteration
   1. this means the advanced user has just hurt him/herself. the data they entered for the topic is not what they really meant to enter. (i.e. "Id" would mutate to the last user to edit, rather than the advanced user him/herself.)
5. it's best to always force keyword exportation so the text is made static, i.e. the user enters the text and it is saved "as entered".
   • In this case TWiki behaves in the manner of "do what I mean, not as I say" and data is always saved as entered

• QED, always forcing -kv is best.

The real reason I'm responding here is because after just updating from cvs, cvs log -r1.46 lib/TWiki.cfg has deleted some lines, and seems to break testenv; the log for the edit mentions this topic as the fix. So this may be a tangent. The removed lines are vars for "lsCmd", "egrepCmd", and "fgrepCmd" which likely has nothing to do with the "coCmd" change required for this enhancement (grin).

My testenv now "complains" (after I merged the v1.46 changes into my TWiki.cfg by deleting the vars):

$egrepCmd: /bin/egrep 
 Note: This is a program TWiki uses for search. 
 Warning: Search program /bin/egrep not found. Check the path. 
$fgrepCmd: /bin/fgrep 
 Note: This is a program TWiki uses for search. 
 Warning: Search program /bin/fgrep not found. Check the path. 
$safeEnvPath: /bin:/usr/bin:/usr/pkg/bin 
 Note: This is used to initialise the PATH variable, and is used to run the 'diff' program used by RCS, as well as to run shell programs such as Bourne shell or 'bash'. 

Note, I run NetBSD, so these pathnames are really:

bash-2.05# which fgrep
/usr/bin/fgrep
bash-2.05# which egrep
/usr/bin/egrep

If egrep & fgrep are no longer used then perhaps testenv needs updating also.

Cheers!

-- JonathanCline - 27 Nov 2002

It's quite possible to paste in a version string from a Perl program that includes ID keywords, without consciously wanting to use the keywords - hence average users can use ID strings (and have used them on TWiki.org). TWiki then corrupts the version string, removing information - so the default is now not do do this. People who don't like this default can change it of course, to -kv or whatever. The default behaviour is also the only behaviour implemented by RcsLite.

Please put any more advanced feature requests in this area in a separate FeatureEnhancementRequest - this is a bugfix.

As for egrep etc - testenv is simply testing your TWiki.cfg, which points to non-existent paths for egrep and co. You should not just update your TWiki.cfg from CVS, since you will need specific settings for your setup. TWiki only uses the PATH variable for RCS's use of diff, and for the shell - everything run explicitly uses a full pathname.

-- RichardDonkin - 27 Nov 2002

Now applied to TWiki.org, so the following is not updated when saved:

$Id fred@example.com 11 Dec 2002$  (wrong keyword)

$Id: ExpandsRcsKeywordsInText.txt,v 1.18 2003/04/03 12:06:11 ChristianFroehler Exp nobody $

$Id: ExpandsRcsKeywordsInText.txt,v 1.18 2003/04/03 12:06:11 ChristianFroehler Exp nobody $

Well, JonathanCline's & RichardDonkin's arguments convinced me; I didn't think of all the diffs and sources pasted in and meant to be literal. Requested RevisionDataVariables.

-- PeterKlausner - 11 Dec 2002

Testing this again on TWiki.org - not working, see Support.ExpandsRcsKeywordsInText.

-- RichardDonkin - 02 Apr 2003

Workaround (See also the discussion on Support.ExpandsRcsKeywordsInText )

This workaround prevents RCS keywords like $Id:$ or $Log:$ being expanded by the Version Control System that Twiki uses.

What it does not do:

• It does not treat the expanding of RCS keywords in newly created topics; the steps described below have to be re-executed for every new topic that contains RCS keywords which shall not be expanded.
• RCS keywords in appended text files are also expanded, which is not addressed by this workaround.

Follow these steps:

• Make a backup of the TWiki installation
• On the server open a terminal window and go to the directory that contains the topics (usually data/Name_Of_Your_Twiki_Web).
• Execute rcs -ko *.txt
• Now RCS keywords will not expanded when saving an existing topic.

-- ChristianFroehler - 03 Apr 2003