Question

Hi

I've just installed the latest TWiki. (Dakar rc1 2006-01-26)

Most of the things works fine, but I've problems with Hungarian characters (it is iso8859-2 e.g. áíűőüöúóé) in WikiWords.

And here it comes: I can use WikiWords with these characters inside, but the statistics page is broken. WikiWords gets truncated at the first of these characters.

• WebStatistics was ignorant of I18N topics, see Bugs:Item1554. Thanks for reporting! -- SteffenPoulsen - 31 Jan 2006

Page moved to messages appears in UTF-8 I can see double characters like: 'ĂĄ'. More topic actions has the same errors.

Some settings with configure:
{UseLocale} true
{Site}{Locale} hu_HU.ISO8859-2
{Site}{LocaleRegexes} true
{Site}{CharSet} ISO8859-2
{Site}{Lang} hu

Thanks for you help!

PS: FYI: This used to work in 2004Sept03 I have just upgraded.

-- CSILLAG Tamás

Environment

-- CSILLAG Tamás - 28 Jan 2006

Answer

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.

Rather strange. I assume you don't have any errors/warnings about locales in configure? Output of configure attached here might be useful.

Might be something to do with EncodeURLsWithUTF8, so try commenting out the line in TWiki.pm that looks like this:

   $fullTopicName = Encode::decode("utf8", $fullTopicName);

This seems to be a regression from CairoRelease. So the other suggestion is to get onto TWikiIRC where the DakarRelease developers hang out, as this would need to be fixed in next day or two to make the release (may already be too late as I think it's in code freeze). Or log a bug against DakarRelease.

-- RichardDonkin - 30 Jan 2006

Well, I read the other topic after posting this question here, where there were problems with chinese characters, so I already tried this. Commenting out this line:

$text = Encode::decode('utf8', $text);

Makes no difference.

It seems to me that these strings in question were encoded twice (but I'm not sure, I'm not an expert in this topic)

Here the output of my configure script

Thanks.

-- CSILLAG Tamás - 30 Jan 2006

Have you had a look at GermanUmlauteOnWindowsServer2003AndWindowsXP? Looks sort of similar. Upgrading or downgrading the CPAN:CGI module might help, as I think recent versions have broken some bits of I18N. Didn't get confirmation from JoachimBlum, so feel free to email him and ask him to update that page saying what version of CGI.pm finally fixed his problem

In particular, CGI.pm Changes file makes clear that 3.06 introduced a 'fix' that makes all UTF-8 bytes into Perl Unicode characters internally, which is quite likely to break things. So 3.05 might be a good bet, or perhaps the latest version if they've made that configurable. Or, if you are into Perl, just try commenting out the line return chr($c) if $] > 5.006;= in utf8_chr in CPAN:CGI::Util. Also, see the last comment in this CGI.pm bug entry.

-- RichardDonkin - 30 Jan 2006

Well, i don't think that this is a CGI.pm related issue (it can be btw). Two days ago the whole setup worked with CairoRelease (04 Sep 2004). However I tried as you suggested, but it did not help.

My problem is similar to the quoted GermanUmlauteOnWindowsServer2003AndWindowsXP, but not the same. The wiki is functional, I mean I can edit topics with these characters (just tried), but weird characters appears in ... I skip that part because I 've just realized that all of this are part of the translated messages??? The InternationalizationFramework ??

My translation is stored in UTF-8 (hu.po), and it is encoded to iso8859-2 when it is sent to the browser, but what happens to placeholders? I mean

%1

I suspect a bug here when these are encoded from one encoding to another and back again.

Thanks, and sorry for my bad english I hope it is still understandable

-- CSILLAG Tamás - 30 Jan 2006

OK - you need to have a look at UserInterfaceInternationalisation and comment there. Or get onto TWikiIRC perhaps at a time when AntonioTerceiro is likely to be there (he's in South America).

-- RichardDonkin - 31 Jan 2006

Also, does the problem go away when you disable user interface internationalisation using configure? That would help diagnose this problem.

-- RichardDonkin - 31 Jan 2006

The disable user interface internationalisation was a great idea! It proved the bug I described above. Disabling the internationalisation solved the problem. (however we need it btw )

-- CSILLAG Tamás - 31 Jan 2006

Oh, man. I tried to comment UserInterfaceInternationalisation and after writing for a while (in the comment box) I realized it is a protected page.

So please do it for me.

-- CSILLAG Tamás - 31 Jan 2006

I didn't think that UserInterfaceInternationalisation was protected, but just create a new BugReport in Codev, e.g. ProblemWithUserInterfaceInternationalisation, or even better at Bugs:WebHome (which is the server used by DakarRelease developers).

You do need to create a TWiki.org userid really... perhaps Codev is protected against TWikiGuest users? It's quite painless, just visit TWikiRegistration.

-- RichardDonkin - 31 Jan 2006

Reported this as Bugs:Item1556. BTW: CSILLAG, we will welcome a hu translation very much if you don't mind releasing it in the wild

-- SteffenPoulsen - 31 Jan 2006

I will try to do my best, but my translation is not complete yet.

-- TamasCsillag - 01 Feb 2006