Bug: No autolink if there are German umlaute in a WikiWord

(Fixed, see below in Followup)

German umlaute break WikiWords. I ran into this problem before (see HowToDealWithGermanUmlaute) and then it was solved by using the TWiki beta 18Dec2003. Our wiki is still running on this beta and everything works fine. I've installed Cairo in a different directory on the same server, the config file settings for locale and use_locale are the same, too. But in Cairo TestUmläute does not become a WikiWord (TestUmläute), whereas [[TestUmläute]] does (TestUmläute). Also, existing topics that have Umlaute in their title are broken as well.

Testenv says:

• $useLocale: 1
• $siteLocale: de_DE.ISO-8859-1
• $siteCharset: iso-8859-1

Our wiki installations can be found at (use "WikiGast" and "gast" as username and password to login):

• http://lenz.uni-koblenz.de/twiki/bin/view/Spielwiese (Bejing release, 18Dec2003, umlaute work fine)
• http://lenz.uni-koblenz.de/newtwiki/bin/view/Spielwiese (Cairo release, same server, same config values, umlaute don't work anymore)

Test case

• TestUmlaute becomes a WikiWord: TestUmlaute
• As does [[TestUmläute]]: TestUmläute
• TestUmläute however does not: TestUmläute

Environment

-- ChristianKohl - 01 Sep 2004

Workaround

I experience this problem as well with danish chars and danish locale settings (eg. "BlåBærGrød" / da_DK.iso885915 ). To work around it, I've set the following vars in TWiki.cfg:

$useLocale = 0;
$localeRegexes = 0;
$upperNational = 'ÆØÅ';
$lowerNational = 'æøå';

This will make the the danish WikiWords work again (substitute ÆØÅ/æøå for other languages). This hint is only meant as a workaround for those who suffer - we need to get the locales to work to get the sorting in place also.

-- SteffenPoulsen - 05 Sep 2004

Follow up

This is related to CyrillicWikiWordError. RichardDonkin is currently busy with other things, could someone look into fixing this? This is a showstopper for non-English sites using TWikiRelease01Sep2004.

-- PeterThoeny - 05 Sep 2004

This might be a suggestion for a fix:

1. Make sure you've installed all the packages needed for UTF conversion. On debian this is done by apt-get install libi18n-charset-perl libunicode-map-perl libunicode-maputf8-perl (it's the perl modules named Unicode::MapUTF8, Unicode::Map, Unicode::Map8 and Jcode you need).
2. Set up the internationalisation variables in TWiki.cfg. For some reason, I can't use da_DK.ISO-8859-15 for $siteLocale without errors (it gives me a lot of "Encode::FB_PERLQQ()"-style errors) .. but just da_DK by itself works great, luckily.
3. Add use locale; to lib/TWiki/Render.pm, line 39 (thanks to VladimirPodryvanov in CyrillicWikiWordError for this one!).

This works for me - I don't need the workaround anymore.

-- SteffenPoulsen - 05 Sep 2004

Wuuuhuu! - After applying the CyrillicTopicNameError fix, I can use da_DK.ISO-8859-15 as well. Honey, I'm hoooome!

-- SteffenPoulsen - 05 Sep 2004

Yes, it works for German as well! Thanks again for the quick response (and solution of course g)!

-- ChristianKohl - 06 Sep 2004

Fix record

Thanks to Steffen and Vladimir. The fix is now in SVN.

Note: To fix TWiki on your own non-English site apply below patch and the one in CyrillicTopicNameError.

Modified: twiki/trunk/lib/TWiki/Render.pm
===================================================================
--- twiki/trunk/lib/TWiki/Render.pm     2004-09-05 04:53:17 UTC (rev 1744)
+++ twiki/trunk/lib/TWiki/Render.pm     2004-09-06 05:40:00 UTC (rev 1745)
@@ -38,6 +38,18 @@
 
 use TWiki qw(:renderflags %regex $TranslationToken);
 
+# ===========================
+# Read the configuration file at compile time in order to set locale
+BEGIN {
+    do "TWiki.cfg";
+
+    # Do a dynamic 'use locale' for this module
+    if( $TWiki::useLocale ) {
+        require locale;
+        import locale ();
+    }
+}
+
 # Globals used in rendering
 use vars qw(
        $isList @listTypes @listElements

-- PeterThoeny - 06 Sep 2004

Thanks to everyone for fixing this. The use locale patch above is something that all modules using the I18N regexes should automatically incorporate so they work in the I18N mode. Some refactoring of TWiki broke this after I finished the EncodeURLsWithUTF8 feature, and nobody (including me) was testing more recent releases for I18N breakage.

As for the Perl modules, if you are using Perl 5.8.x there are no extra modules to be installed (the built-in CPAN:Encode module does everything). If you are on Perl 5.6.x or earlier, you'll need to install the CPAN:Unicode::MapUTF8 module, which should pull in the other ones if you are using the CPAN:CPAN module to install (perl -MCPAN -e shell).

The docs for this feature are at AppendixEncodeURLsWithUTF8 - please fix the docs if they are confusing or missing information, e.g. re the required Perl modules.

-- RichardDonkin - 01 Oct 2004

Another way to get I18N working on 01Sep2004 is to just install the latest TWikiBetaRelease - probably the best option if you are doing a new installation.

In the above patch, it's a bit more efficient to delete the do "TWiki.cfg" - already done in SVN:lib/TWiki.pm. Also fixed in SVN on MAIN.

-- RichardDonkin - 26 Oct 2004

There's a separate issue when it comes to plugins. I don't know how to address this, but many of the plugins doesn't work with I18N topics. For instance, UpdateInfoPlugin won't trigger on updated I18N topics and TocPlugin won't cross-reference I18N topics. (And FindElsewherePlugin won't see I18N words elsewhere - SP).

Not a big issue for looser content, but when it comes to structured content and TWikiApplications, this is kind of a problem. At our place, many of us are still using ascii-7 topic names for this reason.

I might have a go at lifting a few of the plugins to be I18N-compatible, but I'd have no idea where to begin. Richard, can you give a few hints on what to look for? Perhaps there is some kind of guideline that new plugin authors could refer to as well, on I18N issues in plugins?

-- SteffenPoulsen - 26 Oct 2004

Missed this comment first time round, just before I went on holiday... Various TOC bugs have been fixed recently, so try the patches from InternationalisationIssues.

UPDATE: The main thing to do when internationalising plugins is to use the InternationalisationGuidelines. This makes it quite easy to I18N enable your plugins.

-- RichardDonkin - 08 Dec 2004

Apparently this has been fixed. I assume it's merged?

-- CrawfordCurrie - 13 Feb 2005

Yes, was fixed some time ago - original fix was improved upon to avoid re-running TWiki.cfg.

-- RichardDonkin - 14 Feb 2005

Dakar Release - shouldn't it be default to support ISO-8859-15 (and others) in WikiWords?

-- FerdinandGassauer - 29 Jan 2006

The only way to get accented characters in WikiWords is to set the {Site}{Locale} parameter (in DakarRelease terms) to something like de_DE.ISO-8859-15. It's the whole locale setting that ensures that accented characters are accepted in WikiWords - so there's no way to make (say) en_US.ISO-8859-15 accept accented characters by default.

So the answer is no, but it will work well out of the box just by setting this and {UseLocale}.

-- RichardDonkin - 29 Jan 2006