Question

There are different situation:

1. in the Topic Edit and View page, All Chinese Characters Display Normally
2. in the Topic Raw View, All Chinese Characters Display Not Normally
3. in some page, like Recently Changed Topics, Not All Chinese Characters Display Normally

I have tried to set other value at {Site}{Locale}, like zh_CN.UTF-8, zh_CN.gb2312, EUC-CN...But not complete work.

But there is some TWiki site which used Chinese well, like this, http://www.pgsqldb.org/

Environment

-- LiuJia - 17 Nov 2006

Answer

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.

Thanks for the screenshots and for uploading the configure output as requested in SupportGuidelines! This makes it a lot easier to help.

You are recommended to set zh_CN.utf8 for Chinese, or whichever locale is available (see locale -a). Sometimes the .utf8 suffix in your locale setting is not the same as what the browser expects, but utf-8 should work - see {Site}{Charset} in configure to override if the browser expects something different from the locale's charset setting (but note that this is only needed in rare cases, normally the locale setting is enough). See http://www.pgsqldb.org/twiki/bin/testenv for an example of this override, at the end of the page - note they are using an earlier version of TWiki so that testenv is used instead of configure, but the principles are the same.

This test page, which links to http://www.pgsqldb.org/twiki/bin/view/Sandbox/另外一个, is a page with Chinese characters in UTF-8 that display OK in both normal view and Raw view. The fact that this doesn't work in 'Raw' mode may be an I18N bug, but it could also be your setup - we would need to do some TWikiDebugging to find out exactly what's going on. Also, this WebChanges page shows Chinese characters in both page contents and page names.

Since the attached .txt file is valid UTF-8, perhaps TWiki is not setting the headers correctly for your locale in some situations. It would be worth installing the Firefox LiveHttpHeaders extension available at http://livehttpheaders.mozdev.org/ so that you can see what character encoding is set by TWiki in the HTTP headers for (1) normal view and (2) raw view. Or just use wget which is probably included in FreeBSD.

Also, it may be worth turning off the "Auto-detect encoding" feature of Firefox under View | Encoding. Testing with another browser, e.g. Opera, would also be useful.

-- RichardDonkin - 23 Nov 2006 %COMMENT{type="

Thanks for replay. It seems to me that the problem was sloved. Here are some details:

Problem1:
There are some 'question with black skin' and 'right arrow' sign replace some Chinese characters in the 'Topic Raw View'

Solved1:
Add __CGI::header(-charset=>'utf-8');__after line 254 at TWiki/lib/TWiki/UI/View.pm

Changed:

    my $page;
    # Legacy: If the _only_ skin is 'text' it is used like this:
    # http://.../view/Codev/MyTopic?skin=text&contenttype=text/plain&raw=on
    # which shows the topic as plain text; useful for those who want
    # to download plain text for the topic. So when the skin is 'text'
    # we do _not_ want to create a textarea.
    # raw=on&skin=text is deprecated; use raw=text instead.
    if( $raw eq 'text' || $raw eq 'all' || ( $raw && $skin eq 'text' )) {
        # use raw text
        $page = $text;
    } else {
        my @args = ( $session, $webName, $topicName, $meta, $minimalist );

        $session->enterContext( 'header_text' );
        $page = _prepare($start, @args);
        $session->leaveContext( 'header_text' );

        if( $raw ) {
       if ($text) {
         my $p = $session->{prefs};
         CGI::header(-charset=>'utf-8');
         $page .=
      CGI::textarea( -readonly => 'readonly',
                -wrap => 'virtual',
                -rows => $p->getPreferencesValue('EDITBOXHEIGHT'),
                -cols => $p->getPreferencesValue('EDITBOXWIDTH'),
                -style => $p->getPreferencesValue('EDITBOXSTYLE'),
                -default => $text
              );
       } 

Problem2:
There is an 'question with black skin' at the end of sentence, follow an Chinese character

Solved1:
Del a space between '/' at the line 1301 at TWiki/lib/TWiki/Render.pm

Changed:

sub makeTopicSummary {
    my( $this, $theText, $theTopic, $theWeb, $theFlags ) = @_;
    ASSERT($this->isa( 'TWiki::Render')) if DEBUG;
    $theFlags ||= '';

    my $htext = $this->TML2PlainText( $theText, $theWeb, $theTopic, $theFlags);
    $htext =~ s/\n+//g;

    # FIXME I18N: Avoid splitting within multi-byte characters (e.g. EUC-JP
    # encoding) by encoding bytes as Perl UTF-8 characters in Perl 5.8+. 
    # This avoids splitting within a Unicode codepoint (or a UTF-16
    # surrogate pair, which is encoded as a single Perl UTF-8 character),
    # but we ideally need to avoid splitting closely related Unicode codepoints.
    # Specifically, this means Unicode combining character sequences (e.g.
    # letters and accents) - might be better to split on word boundary if
    # possible.

    # limit to n chars
    my $nchar = $theFlags;
    unless( $nchar =~ s/^.*?([0-9]+).*$/$1/ ) {
        $nchar = $TMLTRUNC;
    }
    $nchar = $MINTRUNC if( $nchar < $MINTRUNC );
    $htext =~ s/^(.{$nchar}.*?)($TWiki::regex{mixedAlphaNumRegex}).*$/$1$2 \.\.\./s;

    # newline conversion to permit embedding in TWiki tables
    $htext =~ s/\s+/ /g;

    return $this->protectPlainText( $htext );
}

'Topic Raw View' miss some Chinese characters also display at another (testing)site (maybe is a new TWiki version)
links: http://pbx1.astercon.com/

'Question with black skin' sign also display at another site (maybe is TWiki version: 04 Sep 2004 Rev: 1742 )
links: http://www.pgsqldb.org/

-- LiuJia - 23 Nov 2006