Question

If all characters in the name of a form field are non-ASCII alphabetic (Greek ISO-8859-7, to be specific), the form behaves strangely. Specifically:

1. I create a form which has several form fields of which all characters in the name are Greek.
2. I add the form to a topic. The form shows correctly in edit mode. I fill in the fields.
3. When the topic is previewed/saved, successive all-greek fields are collapsed as follows:
   • Only the last of a series of successive all-greek fields shows
   • Its value is the value I had filled in for the first of the series of successive all-greek fields
   • The other values of the series are lost.
4. Upon editing again, the form shows correctly in edit mode, but field values have been lost; series of successive all-greek fields all have the value that had been entered for the first one of them.

You may be able to see it illustrated at http://www.itia.ntua.gr/twiki/bin/view/Sandbox/TestForInternationalFormFields. If you want to experiment, you'll have to register. I'm not sure this is possible, but you can try.

The apache error log shows many messages like this:

[Tue Mar 30 12:41:17 2004] null: Use of uninitialized value in string eq at ../lib/TWiki/Meta.pm line 57.

(Most of the times the problem is in line 57; sometimes, namely when clicking "Edit", the same error message but at line 104 shows).

I uploaded the output of testenv and my TWiki.cfg.

Environment

TWiki version:	TWikiRelease01Feb2003
TWiki plugins:	DefaultPlugin, EmptyPlugin, InterwikiPlugin, ConditionalPlugin, TablePlugin
Server OS:	Debian 3.0 i386 with precompiled kernel 2.4.18-1-686
Web server:	Apache 1.3.26-0woody3 with mod_perl 1.26-3.0woody1
Perl version:	5.6.1 (prepackaged 5.6.1-8.6)
Client OS:	Irrelevant; problem shows in both Win and Linux
Web Browser:	Mozilla Firefox 0.8, but problem has also shown in previous versions

-- AntoniosChristofides - 30 Mar 2004

Answer

Thanks for providing the info and the test URL - I made a change there as TWikiGuest.

The crux of this issue is the 'fields with all-Greek names', although possibly Latin values. Given that, there is a suspicious line of code in the _cleanField sub in CVS:lib/TWiki/Form.pm that would strip out any Greek characters from field names, causing all-Greek field names to have the name '' (zero-length string):

   $text =~ s/[^A-Za-z0-9_\.]//go; # Need dot for web.topic

Try changing this to the following (not tested):

   $text =~ s/[^\w\.]//go; # Need dot for web.topic

And insert the following right at top of the file:

# 'Use locale' for internationalisation of Perl sorting and searching -
# main locale settings are done in TWiki::setupLocale
BEGIN {
    # Do a dynamic 'use locale' for this module
    if( $TWiki::useLocale ) {
        require locale;
        import locale ();
    }
}

This should ensure that international characters are not stripped out of the field names. There may be another issue at line 544 as well.

See CVS:lib/TWiki/Search.pm for example of how internationalisation was done on a TWiki/*.pm module, as long as you follow this pattern you should have few problems - the basic idea is to use the TWiki regexes instead of \w since the latter works on only some versions of Perl (5.6.1 is OK, so it should be fine for your installation).

Thanks for logging this, I'll work on a real fix when I get time.

-- RichardDonkin - 30 Mar 2004

Thanks, I'm certain I've done this before at some other place of the code (not for forms), although I couldn't find where exactly. Unfortunately \w doesn't work (maybe it would work if I set LC_CTYPE at the script that starts apache on boot, but that's not a good solution either), so I used \S, which is, for me, a good workaround. It was necessary to change line 544 as well.

-- AntoniosChristofides - 30 Mar 2004

Glad that worked - did you do the 'use locale' stuff as well? If so, it should have worked with \w, but it may need some more work - in any case, the regexes from CVS:lib/TWiki.pm (setupRegexes sub) are the best option so it works on all versions of Perl.

-- RichardDonkin - 31 Mar 2004

Richard, is there a followup in Codev on this fix?

-- PeterThoeny - 02 Jun 2004

No, I didn't get time to really investigate and follow this up - however, the key part of the problem is that cleanField and similar are 'cleaning up' valid I18N characters in field names. The 'real' fix should use the various Perl variables defined in the regex setup routine in TWiki.pm.

-- RichardDonkin - 07 Jun 2004

Now fixed in DEVELOP SVN, see Codev.InternationalCharactersInFormFields for details.

-- RichardDonkin - 12 Dec 2004