Bug: Case insensitive search doesn't work with 8-bit characters

Filed by Stefan Lindmark, stefan /// at /// sun.com

Test case

• Several pages contains the word "Våra"
• Search for "våra" finds them
• Search for "VÅRA" doesn't find them
• Name "Örjan" not found in search for "ö"
• Status the same with or without my quick&dirty i18n fix described in InternationalCharactersInWikiWords

Suggested fix!

The problem is caused by fgrep using LC_COLLATE and LC_CTYPE to determine how mixed cases are handled.

The solution:

• Introduce a TWiki variable for character set selection, e.g. en_US.ISO8859-15
• Set the variable somewhere suiteable using

$ENV{LC_CTYPE}=$theCharSet;
$ENV{LC_COLLATE}=$theCharSet;

Tested by adding the following two lines to TWiki.cfg:

$ENV{LC_CTYPE}="en_US.ISO8859-15";
$ENV{LC_COLLATE}="en_US.ISO8859-15";

See also: LocalizationIsNeeded

Environment

-- TWikiGuest - 30 Jan 2002

Sometimes I wonder if anybody reads these topics. Feedback on this has been nil, even though I personally think it's a pretty good stop-gap until TWiki goes UTF8 or implements universal language support in any other way.

-- StefanLindmark - 02 Oct 2002

I did read this both times ... The trouble is that i18n is a big area that I am not too familiar with, and I'm a bit reluctant to make piecemeal changes.

However, if you can produce a patch file as per the PatchGuidelines, with changes to TWiki.cfg and TWiki.pm as required, that would help, and since it is a small change could probably be put into the core easily enough. The TWiki.cfg file should have only the selected charset variable, the env var stuff should go where other env vars are set (search for $safeEnvPath). Also, wouldn't it be better to just set the LC_ALL variable instead of the two LC_* variables?

An alternative would be to just set the environment variables required in the web server environment - this is easily done as in WindowsInstallCookbook for Apache, but is harder to set up.

-- RichardDonkin - 10 Nov 2002

Fix record

Now fixed in TWikiAlphaRelease - see test case running the new code, and InternationalisationEnhancements for more info.

-- RichardDonkin - 30 Nov 2002