Question

Does anyone have a (working) solution on how to treat German Umlaute in topic titles? I've read the Internationalization issues - but asking the user to change his IE settings does not work for us. I either need to apply a patch to the TWiki source that changes German Umlaute when they are entered (i.e. from ä to ae, ö to oe ...) or does not allow the user to use them in the first place. There was a patch from Harald Wilhelmi posted in InternationalCharactersInWikiWords but it doesn't work for the current TWiki release which we are using (TWiki01Feb2003). Could anyone perhaps point out to me what I would have to change?

Environment

TWiki version:	TWikiRelease01Feb2003
TWiki plugins:	lots
Server OS:	SuSE Linux
Web server:	Apache 1.x
Perl version:
Client OS:	various
Web Browser:	various

-- ChristianKohl - 10 Dec 2003

Answer

I have some working code in TWikiAlphaRelease for UTF-8 support for URLs, which avoids the need to reconfigure browsers. This is running on my site's I18N test web - see EncodeURLsWithUTF8 for the details and ProposedUTF8SupportForI18N for the overall plan for UTF-8. It works pretty well for ISO-8859-1 sites, but more work is needed for sites with other encodings; also, attachments to topics with I18N names don't work, though there's a fairly simple fix based on =%INTURLENCODE% of the attachment URL.

This should avoid the need to restrict use of umlauts or convert them to the 'ae' format, since browser setup is no longer needed.

Please comment here or on EncodeURLsWithUTF8 if this doesn't work for you - the main limit on developing this is time, but the second one is lack of people who are willing to test this code, so testing the TWikiAlphaRelease would be very helpful (it's usually quite stable, as it runs TWiki.org as well as my own site). Also, if you can code in Perl I'd be willing to accept patches since I may get some time over the Xmas period to code anyway.

Getting the I18N code to a point where it meets most requirements is a key goal of mine, and lack of UTF-8 URL support is an issue for some people.

-- RichardDonkin - 10 Dec 2003

Follow-Up

Thanks for the quick reply, Richard! I just downloaded the Alpha release and I hope I'll find the time to test it next week. I'll post the results here as requested.

-- ChristianKohl - 11 Dec 2003

I've run a short test with the latest TWikiAlphaRelease (TWiki20031215alpha.tar.gz).

• Our server uses Perl 5.8.0.
• Site locale set in TWiki.cfg is: de_DE.ISO-8859-1

I made the following observations:

• WikiWords containing German Umlaute (GU) are not automatically linked.
  • Example: SchönerStrand becomes SchönerStrand.
• WikiWords with a GU as last character are automatically linked ignoring the last char.
  • Example: TestUmlautä becomes TestUmlautä
• Topics containing GUs have to be linked using the [[...]] syntax then.
• When using the [[...]] syntax GUs are working fine. I checked Mozilla 1.4 for Linux and WinXP as well as IE 6 for WinXP. No problems with any of them.

So it looks like the new code will definitely improve GU-usability. But why are WikiWords containing GUs are not automatically linked? That would IMHO be the least confusing solution for the user. Especially the "chopping off" of GUs at the end of a WikiWord might lead to chaos on German speaking sites when users start complaining about "lost" topics.

-- ChristianKohl - 16 Dec 2003

Just got back from a trip... Thanks for testing this - seems that the UTF-8 URL code is working OK.

I think you may have a setup problem regarding the auto-linking of WikiWords - this is working fine on http://donkin.org/ and was a significant part of the I18N work last year. Please attach the HTML output of testenv, and also your twiki.cfg file (for the Internationalisation section).

Also, the output of locale -a is important to check that the locale listed does in fact exist on your system. If testenv can't switch locale correctly to de_DE.ISO-8859-1, there may be a problem with your Linux or Perl locale setup.

-- RichardDonkin - 18 Dec 2003

Hello ... I found the problem. Hit me, please. I forgot to set use_locale in the TWiki.cfg of the alpha-install. I changed this and now it works fine. No problems with WikiWords anymore. Thanks for reminding me of the cfg-file ... I was about to attach it when I noticed my mistake.
There's just one more question I got now: How stable is the alpha release? It's really hard to follow the changes made since I'm not allowed to view the changelog ChangeLog. Or would it be possible to patch your code into the Bejing release? The main reason I'm asking this is because I had to make a lot of changes to the templates and also some to the scripts to improve usability on our side. If I was going to use the alpha release I would have to make all these changes again and it wouldn't be nice to find out after a couple of weeks that it's not stable.

-- ChristianKohl - 19 Dec 2003

Glad it's working OK, let us know how you get on.

The alpha release is normally fairly stable, but see CVS:bin and CVS:lib (or the whole ViewCVS web view) for the full change history (that ChangeLog page points to a special server administrated by MartinCleaver, not the main CVS logs, which are open to all). You might also want to browse the TWikiDevMailingList archives, which show the changes from your 15 Dec alpha to the 19 Dec beta.

In fact, the most recent TWikiBetaRelease has the same UTF-8 URL code as the alpha you used (and virtually the same code I would guess), so you might want to use that. I haven't had any problems with the UTF-8 URL code since Sept using various browsers, apart from attachments to pages with umlauts etc in the name, so that part is also quite stable.

If you use the TWikiBetaRelease you should be fine - by the way, the user and password for download is your WikiName and password, or just TWikiGuest and 'guest'.

Also, I noticed your TWiki setup linked from ChristianKohl has some translated pages (at least the registration pages) - you might want to collaborate with PatrickHansmeier, who is doing a GermanTranslation of TWiki pages.

-- RichardDonkin - 19 Dec 2003

Thanks Richard, I'll use the BetaRelease then and test it "out in the open" for a few weeks.

As for the German translation: I'll check out the efforts made by Patrick so far and see what I can contribute. The problem is that some of the translations are tailored to our potential user group and are therefore not 1:1 translations from the original TWiki content. But thanks a lot for pointing out all these pages to me. -- ChristianKohl - 19 Dec 2003

I've been running the Dec18-beta-version for two weeks now - no problems so far. Umlaute are working fine. -- ChristianKohl - 07 Jan 2004

Glad this is working for you. In retrospect, UTF-8 URL encoding support is important to wide adoption of TWiki I18N since few people are happy to reconfigure their browsers.

-- RichardDonkin - 13 Jan 2004

UPDATE: In Cairo release you cannot use umlaute in WikiWords. Dunno why. See GermanUmlauteBreakWikiWords

-- ChristianKohl - 03 Sep 2004

UPDATE: The patches posted in GermanUmlauteBreakWikiWords fix the issue!

-- ChristianKohl - 06 Sep 2004