Bug: Search does not work with bad TWiki names

Twiki unfortunately does not prevent creating TWiki pages with bad names (one might get a warning, but can create the page anyway). This can cause a fatal failure in the search functionality:

If you create a topic named Test() and then try WebSearch, this will entirely fail due to a shell error.

The apache 2 logfile APACHE_HOME/log/error_log then contains the following lines:

sh: -c: line 1: syntax error near unexpected token `Test()'
sh: -c: line 1: `/bin/fgrep -i -l -- 'something . Very / unLikelyTo + search - for ;-)' Test().txt TestTopic1.txt 
TestTopic9.txt WebChanges.txt WebHome.txt WebIndex.txt WebLeftBar.txt WebNotify.txt WebPreferences.txt 
WebRss.txt WebSearch.txt WebSearchAdvanced.txt WebStatistics.txt WebTopicList.txt'

TWiki should either

• prevent the creation of such pages or
• escape problematic characters when assembling the shell command

Test case

Go to Sandbox and create a page named Test() and save it.

Run WebSearch in this web and watch the Apache2 logfiles.

You will not get any search results.

Environment

-- GuidoOstkamp - 08 Mar 2005

Impact and Available Solutions

Follow up

Fix record

Discussion

This is not an issue in DevelopBranch, due to the use of the sandbox for all command-line activities.

-- CrawfordCurrie - 09 Mar 2005

The error log output caused a search result on Codev.WebSearch. I've added spaces in the 'something.Very/unLikelyTo...' line.

You see - it's not that very unlikely.

-- ArthurClemens - 15 Apr 2005

I'm afraid I don't understand Crawford's comment about the use of the sandbox, but I'm watching my error logs on my web server and this is one of the ones I see a lot. The problem in my case is that people are creating attachment files whose names contain whitespace; when the grep command is invoked that fails. I'll attach a simple patch that fixes the problem unless your filename contains a single-quote character (').

To me this entire method of invoking the grep command is simply fatally flawed. Not only does it suffer from an extremely difficult quoting problem (especially when you consider portability!!) which is not just a functional issue but also a security issue, but it's slow as well.

The Right Thing To Do here is to write a Perl function that invokes the command directly, the way system() does when you pass it multiple arguments, but that replaces `` (grabs the output). I'd be very surprised if a module like that doesn't already exist but if not I have Perl code I'd be happy to donate that will do the job.

-- PaulSmith - 14 Jun 2005

I think it may be related to SearchedTopicnameCannotContainNonAlphanumChars :

The problem comes from _makeTopicPattern subroutine in lib/TWiki/Search.pm, which does not deal correctly topics with bad Twiki names.

See SearchedTopicnameCannotContainNonAlphanumChars for more details and workaround.

-- NicolasRaibaut - 27 Jan 2006