Bug: Topic save error with Topics containing space

When creating a topic containing spaces, I get this error:

Topic save error 
During save of file Sandbox.Fis er godt an error was found by the version control system. Please notify your TWiki administrator. 
/usr/bin/ci -q -l -m'none' -t-none -w'PeterMorch' /home/groups/t/tw/twiki/p/data/Sandbox/Fis er godt.txt 2>&1
ci: /home/groups/t/tw/twiki/p/data/Sandbox/Fis: No such file or directory
ci: er,v: Permission denied
ci: godt.txt,v: Permission denied

(I tried this on twiki.org, and there is now presumably a '/home/groups/t/tw/twiki/p/data/Sandbox/Fis er godt.lock' and a /home/groups/t/tw/twiki/p/data/Sandbox/Fis er godt.txt' file lying around now...)

OK so it is discouraged to use non WikiWord-s for topics, but not impossible. And when someone does that we croak on it. I tried a pseudo-patch, putting '' around the filename. But it doesn't do the job properly...

TWiki.cfg has lines like this in it:

    coCmd         => "$rcsDir/co $rcsArg -q -p%REVISION% $keywordMode %FILENAME% $endRcsCmd",

If %FILENAME% contains spaces, that gives the problems... I tried naively replacing %FILENAME% with '%FILENAME%' (or rather $cmdQuote%FILENAME%$cmdQuote) but that messes up because

    ciDateCmd     => "$rcsDir/ci -l $rcsArg -q -mnone -t-none -d$cmdQuote%DATE%$cmdQuote -w$cmdQuote%USERNAME%$cmdQuote %FILENAME% $endRcsCmd",

and in RcsWrap.pm we find this code:

    $cmd =~ s/%FILENAME%/$file $rcsFile/;

So it is a bit of a messy thing, this "Topic contains spaces" thingy. I can see the next step, putting in $cmdQuote there too, but I can also see this snowballing. For now, I'll just avoid spaces in the topic name. But if a user forcably creates such a topic, it is left dangling in a half-created state....

Maybe an easier solution is to properly refuse to create such topics. I don't know. My experience with the code is very limited.

Peter

Test case

Go to the Sandbox. In the "Go" field, type in e.g. "Topic With Space". Hit enter. that brings up the page "NOTE: This Wiki topic does not exist". There is a warning saying that this is not a WikiWord and therefore not recommended as a topic name. But if you hit the "Create" link in the bottom of the page, it gets created nonetheless. The error occurs when you hit Preview and then Save for the new topic.

I'm looking into making a modification to MailToTWikiAddOn, where I'd like to use TWiki to do an archive of a mailing list, where "any subject goes". And so I was thinking of removing any leading "Re:" or "FWD:" and then use the rest of the subject as the name of the subject. mailtotiki calls TWiki::Store::saveTopic(), and when the $topic contains space, the error was generated. That was how I first discovered it. So I think any call to TWiki::Store::saveTopic() where $topic =~ / / will do. For now, my MailToTWikiAddOn derivative will just basically { $topic = $subject ; $topic =~ s/ //g; } (and remove special chars and do other stuff...).

PeterMorch - 13 May 2003

Environment

TWiki version:	01 Feb 2003
TWiki plugins:
Server OS:	SuSE Linux 7.2
Web server:	Apache 1.3
Perl version:	5.6
Client OS:	Win XP
Web Browser:	IE 6.0

-- PeterMorch - 12 May 2003

Follow up

This was reported in TopiSaveErrorWithNonWikiWord, copied over here. Better error checking needs to be done to prevent topic saves if they contain spaces.

Question from PeterThoeny - 12 May 2003 : PeterMorch: How did you create this topic?

Answer from PeterMorch - 13 May 2003: I've added text to the "Test Case" section.

-- PeterThoeny

What about replacing space in the filename with backslash space ? Perhaps that would work betting that quoting in the file to rcsfile transformation.

-- KevinWalker - 16 May 2003

What about using underscore '_' instead of space? If WikiWord is an underlined link, looks the same.

-- PeterMasiar - 17 May 2003

Just as a note, URLs with spaces in are illegal as far as the HTTP spec is concerned so this bug if anything is just a bug regarding how the complaint to the user should be made. (ie check for topic name with space in, and send them an "oops" instead)

-- MichaelSparks - 20 Jul 2003

I vote for changing spaces to underscores, or just plain eliding them. Please do not provide a numeric conversion that makes the URL unreadable and untypable: in fact, TWiki not doing this was one of the big reasons I chose twiki over MoinMoin and Zwiki.

-- AndyGlew - 11 Aug 2003

Without hesitating I would vote for using underscores. People that are used to WikiNotation (geeks or programmers) would not have entered a space in the name - they know Wiki and they know HTTP. So, according to the person that types a space, the space is meant to be there. It separates two distinct words in a page title, like in New subject. Making this NewSubject throws the person directly back into geekydom, and mentally separates him from the wiki.

Naturally, TWiki would need to support underscores as Wiki_word separators. Then they could be rendered as spaces, and the person would never see the difference: "Hey, that's what I typed!".

-- ArthurClemens - 11 Aug 2003

Arthur, as always, very good argument. The question is backward compatibility if we relax the WikiWord syntax to allow underscores. Many programmers are use underscores in names, so code descriptions in a TWiki topic would show unexpected links. This is not an issue for non-programmers. Possibly a configuration switch?

-- PeterThoeny - 11 Aug 2003

I concur.

Such a feature would allow people to type spaces (which they expect) and for the system to change it in an obvious way.

This also saves awkwardness when trying to respace terms such as McGill and TWiki.

The only respect in which I differ with Arthur is for single words; Arthur prefers them to be designated with a trailing _ (e.g. Ontology_) whereas I prefer them implemented with a leading '.', hence .Ontology; there is a discussion about this at SingletonWikiWordPlugin

-- MartinCleaver - 12 Aug 2003

There is also discussion about it at PatternSkinDev

-- ArthurClemens - 12 Aug 2003

Related to this bug: when I make a link using double square brackets containing, say, "Machine : Blah", and then click on the question mark to edit the new page, then "Machine : Blah" gets rewritten to "Machine Blah" for the topic name of the new page. I then experience the same problem saving a topic whose name contains a space.

-- EdParcell - 13 Aug 2003

PeterThoeny:

• The question is backward compatibility if we relax the WikiWord syntax to allow underscores. Many programmers are use underscores in names, so code descriptions in a TWiki topic would show unexpected links. This is not an issue for non-programmers. Possibly a configuration switch?

Better a per page type - possibly in the meta info. (This way I can migrate my wiki to the better format.)

-- AndyGlew - 14 Aug 2003

If we want to support bracketed notation, like Wikipedia does (for example: [[bracketed notation]]), the number of changes are not that big:

1. The save script has to replace spaces with underscores, and preferably to urlencode other illegal html characters
   • The idea to use underscores is mainly for aesthetic reasons: %20 does look so bad in a url.
2. The create page should not give a warning that the topic name is not a WikiWord
3. Same for the rename script

As long as you keep using brackets to have links, there is not a problem with twiki emphasis markup. So the line

This is an italic line with a bracketed link: [[bracketed_notation_test]]

won't give any problems. Single topic titles can be represented with [[single]]. Topics that are named this way show up normally in Changes and search results.

The underscores in the topic titles can be removed with a link beautifier plugin, so the topic link will just look like bracketed notation test.

BTW, this solution has nothing to do with UnderscoreWikiWords.

-- ArthurClemens - 03 Dec 2003

I've seen suggestion somewhere that tilde ~ is also a character allowed in URL and file name, possibly good replacement for space if we want to use _ for italics only.

-- PeterMasiar - 04 Dec 2003

As I have written above, there is absolutely no conflict between using _ for italics and using _ in bracket notation, at least not technically. I wonder if there is a conceptual problem in using _ in two different contexts.

-- ArthurClemens - 04 Dec 2003

When attempting to create a topic with a space, the topic does get created, despite the Topic save error at the top of this topic. The topic is listed in the WebIndex with a %20 encoded space, but does not show up in the WebChanges list.

Added: to be more precise, it does not show up in the changes page, but is listed on the search results page when doing a search.

-- ArthurClemens - 08 Jan 2004

Specification

• When handling topics, spaces in a topic name should be converted to underscores.
  • When a new topic name is entered, spaces should be converted to underscores
  • When a topic name is read in a double bracket, [[Topic name]] should be read as [[Topic_name]]

-- ArthurClemens - 20 Apr 2004

Fix record

We can split up the problem in 4 separate issues:

1. Add underscores to the possible WikiNames regex.
2. Change the save script so that spaces are converted to underscores
3. Make the bracket link [[Topic name with space]] equal to [[Topic_name_with_space]]
4. Render Topic_name_with_space with spaces like Topic name with space to support search engines and common readability.

Fix 1: Add underscores to the WikiNames regex

The current wikiWordRegex is:

 [<nop>[:upper:]]+[<nop>[:lower:]]+[<nop>[:upper:]]+[<nop>[:lower:][:digit:]]* 

The proposed underscoreWikiWordRegex is:

 [<nop>[:upper:][:lower:][:digit:][_]]+[_]+[<nop>[:upper:][:lower:][:digit:]]* 

Is it trivial to combine these to:

 [<nop>[:upper:]]+[<nop>[:lower:]]+[<nop>[:upper:]]+[<nop>[:lower:][:digit:]]*|[<nop>[:upper:][:lower:][:digit:][_]]+[_]+[<nop>[:upper:][:lower:][:digit:]]* 

Note the combining pipe sign. Yes, I've been taken regex lessons

The regex variable in TWikiDotPm (line 586 or so) then becomes:

$regex{wikiWordRegex} = qr/[$regex{upperAlpha}]+[$regex{lowerAlpha}]+[$regex{upperAlpha}]+[$regex{mixedAlphaNum}]*|[$regex{mixedAlphaNum}_]+_[$regex{mixedAlphaNum}]*/;

And voila! Underscores are working. The exclamation mark to nop a word link also just works.

Update:
Combined syntax such as in AsDoc_mx_data_kinds_Data gives problems: only the first part of the word (AsDoc) is recognized as possible wiki word. Better swap the 2 parts in wikiWordRegex:

$regex{wikiWordRegex} = qr/[$regex{mixedAlphaNum}_]+_[$regex{mixedAlphaNum}]*|[$regex{upperAlpha}]+[$regex{lowerAlpha}]+[$regex{upperAlpha}]+[$regex{mixedAlphaNum}]*/;

This makes links like mx_data_kinds_Data_AsDoc also possible.

-- ArthurClemens - 22 May 2004

-- ArthurClemens - 21 May 2004

Fix 2: Change the save script so that spaces are converted to underscores (partial fix?)

In View.pm, line 128 just before

if( &TWiki::isWikiName( $topic ) || &TWiki::isAbbrev( $topic ) ) {

add:

   $topic =~ s/ /_/g;
   $topic =~ s/%20/_/g;

Now in WebTopicViewTemplate the spaces are changed to underscores, and since the new underscore regex is working you won't get the error page WebTopicNonWikiTemplate.

For automatically creating topics I presume more changes are needed.

-- ArthurClemens - 21 May 2004

Proposed patch

Enclosed patch fixes this problem with latest beta. I also fixed some other related problems:

• topic with spaces can't be found with search

• creating new topics with clicking on [[]] link attempts to convert it to WikiWord. Should TWiki mangle new topic names, it must do it consistently across all ways a new topic can be created!

As far as I tested it, with the patch should TWiki be "space-transparent". If it still isn't, please complain.

It does not follow specifications above, it just preserves the space as is. On the other side, the patch is quite simple. Code for automatically converting between topic name and topic_name would be IMHO more scattered around and bug-prone.

-- JurajVariny - 23 May 2004

What now? Should I keep up the patch, or instead it will be solved more sophisticated way (like UnderscoreWikiWords) in CairoRelease ?

If some my claims (above and in WikiWordDiscuss, SpacesInTopicNamesIssue) were considered outrageous, I apologize myself. They were meant to provoke discussion. But nobody reacted during two weeks...really nobody is interested? Or maybe is it that the spaced wiki words problem isn't discussed in any central place, but in lots of topics:

Search: Spaced

AddNoEncodeParameterToSpacedTopic, AttachmentsWithSpacedWikiFileNames, RenameDoesNotCorrectSpacedTopics, SpacedOutTWikiWords, SpacedTopicNameFormattedSearch, SpacedTopicVariable, WhyDoLinesComeOutDoubleSpaced,

Search: Spaces

RenameTopicAllowsSpaces, SecurityPatchBreaksSpacesInFilenames, SpacesInTopicNamesIssue, TagMePluginWithMultipleTagNamespaces, TmplDefStripsLeadingAndTrailingSpaces, TwoSpacesAfterASentence, WikiWordsWithSpacesBreakPredictability, WikiWordswithSpaces,

It's only rough estimate showing some unrelated topics.

-- JurajVariny - 06 Jun 2004

Sorry - I've been concentrating on completing the tasks assigned to me, so i hadn't noticed this patch (and I assumed that someone more interested in spaced wikiwords would take care of this. At the moment I am happy to commit this patch (or a modification of it (the "my cmdQuote ="'";" bit i assume was test code that you mean to remove)) as it is a good bugfix.

do you have any other changes that you would like to see made to this patch? (or any more test results)?

-- SvenDowideit - 03 Jul 2004

hey Juraj - I've done a quick bit of testing, and so far its great! it even finds some space stuff though its at the end of the search list (weird - i expected lower case preceded upper)

is there any chance that you could look at UploadMungesFilenames and see if you can fix it too ?

could someone else please have a go too?

-- SvenDowideit - 11 Jul 2004

If I won't have time to do UploadMungesFilenames, I will at least try to document this. Where? Should I just change the TextFormattingRules for example?

-- JurajVariny - 12 Jul 2004

yep, TextFormattingRules is a good example. unfortuanatly my familiarity with the documentation is pitiful, otherwise, Where-ever else you think it reasonable would be a huge help there might be some interest amongst the internationalisatoin topics too ..

Thanks for any and all help you give - we really do appreciate it, its just hard to keep track of everything at the moment.

-- SvenDowideit - 14 Jul 2004

This spec needs to be looked at into more details. Possibly defer to DakarRelease.

An easy and consistent fix would be to adjust the name the same as the link part in the square bracket links, e.g. from foo bar to FooBar. This could be done for CairoRelease.

-- PeterThoeny - 15 Jul 2004

from foo bar to =FooBar=

This would be an ugly fix. I was hoping for a more usable solution.

This spec needs to be looked at into more details

Could you elaborate on this? Because the SpecProgress indicates 100% at this moment.

-- ArthurClemens - 18 Jul 2004

Sorry Juraj I'm going to have to defer this for a few weeks to Dakar - I haven't found the time to make sure that I'm happy with all the deatils (what i did see looked good though)

If anyone has the energy, please refactor this into multiple topics to reflect the difference between this bug report and the spawned NewFeatureProposals

-- SvenDowideit - 22 Jul 2004

Problems that would the patch introduce (if any), would be IMHO much less annoying than the problems it solves. If no one worked out specs to make you fully happy in 2 months, then nobody ever will. It would be better to put it into the alpha and watch out for any real problems.

I will likely keep the patch. Topic names munging is in my view a bug, or at least unnecessary "feature" that is impossible to be done well, especially with regard to i18n. The sooner we abandon it, the better.

-- JurajVariny - 31 Jul 2004

Hi, I will probably have time to upgrade/test the patch to Cairo. But can anyone from CoreTeam promise whether it can go through into main tree sometime? I'm interested into removing as many constraints for WikiWord implied by code as possible. And adding the same constraints as fully configurable regex value.

Please don't say again that "WikiWord should never contain spaces nor punctuation, that isn't the Right Thing". I don't have power to learn/enforce anything to my colleagues, and if they can't just use topic names with spaces, they say "it doesn't work" or "it's toooo complicated". Simply, they are used to PHP Nuke, but for the information I maintain (and that I'm trying to put my colleagues hands on) is TWiki much more suitable.

-- JurajVariny - 30 Jan 2005

Since I added spaces to the $cfg{NameFilter} and we are now using the Sandbox everywhere, this should not longer be a problem - it is not possible to have topic names with spaces. If someone wants to put the effort in to make them possible, then be my guest. For now, this is non-critical.

-- CrawfordCurrie - 06 Mar 2005

Related problems affecting Cairo are: TopicSaveErrorWithTopicsContainingSpace, UploadMungesFilenames