Feature Proposals » TWiki to run under the current perl version

Motivation

This is a recurring action, and right now the topic is "Perl 5.24.1". While it is generally a good idea to check that TWiki works with up-to-date Perl, it gets somewhat urgent when this Perl version is found in the wild.

Description and Documentation

The most notable difference in Perl 5.24 is that CGI.pm is no longer part of the Perl core. It is, of course, available as a package from CPAN and from Debian (libcgi-pm-perl), but both of them come in versions which are not compatible with TWiki. As we have learned, CgiContrib is not a complete solution to the problem.

Examples

Right now, Strawberry Perl is available with Perl 5.24, and within 2017 I'd expect the release of Debian Stretch, which ships with Perl 5.24.

Impact

Without some action, TWiki can't be installed on Debian Stretch and will not work with current Strawberry Perl.

Implementation

I'd hope that the effort is rather small. One known issue is the early invocation of CGI::Carp in the TWikiScripts which is not caught properly by CgiContrib.

The plan:

• Run TWiki from SVN using Perl 5.24 and the PSGI Engine to check for other pitfalls. First results:
  • Basic viewing and editing just works, because PSGI avoids the CGI::Carp problem.
  • configure breaks with Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/}{ <-- HERE / at lib/TWiki/Configure/UI.pm line 100. (and perhaps more to come). This is "only" a warning, but in configure, every warning raises a fatal error (which is a good thing).
• Run the test suite under 5.24 (CPAN:App::perlbrew to the rescue)
  • Doing so, it turns out that there are plenty of this unescaped left braces in TWiki regexes. Unfortunately the TWiki syntax %FOO{ is often examined by a regex /\%FOO{.../ which should in Perl 5.24 be written as /\%FOO\{.../. The escaped version works in any Perl version back to 5.8, so we're safe to change it though it is tedious. -- Harald Jörg - 2017-03-25
  • The count for the unescaped left brace issue is about a dozen in core, and about 200 in plugins. Every request causes several warnings to the web server error log, for persistent interpreters there's just one warning per module. Fortunately, the fix can be applied per program: Write a test script, parse the resulting warnings which are pretty precise about files and lines, and fix these lines. Patch available (2455 lines). -- haj 2017-04-01
• Finally, of course, fix the CGI::Carp issue.
  • Two news on that item: First, the fix is a small one, albeit a bit ugly. It is not sufficient to simply move TWiki's lib magic. We must also avoid to use the module within the BEGIN block, because use outruns require. An example diff for view is attached. Second, I discovered by accident that TWiki 6.0.2 will work with Perl 5.26 if both the system's CGI (from CPAN or Linux distribution) and TWiki's CgiContrib are installed: In that case, CGI::Carp is taken from the system (before TWiki adds its own libraries), and CGI later on from CgiContrib. I caught the system library by installing CGI::Session which depends on CGI. I would prefer not to rely on such a construction. -- haj 2017-04-01

-- Contributors: Harald Jörg - 2017-03-21

Discussion

With CGI::Carp, I don't think moving "require 'setlib.cfg';" to the beginning of the BEGIN block of scripts in bin/* is a good idea. Because setlib.cfg reads LocalLib.cfg, which can be big -- mine is 194 lines long. Problems in LocalLib.cfg should be displayed, which is why CGI::Carp::confess or Carp::confess was set to $SIG{__DIE__} at the beginning of the BEGIN block originally.

How about incorporating CGI.pm and CGI/*.pm into TWiki core rather than having CgiContrib? And then split setlib.cfg into two: settwikilib.cfg and setlib.cfg so that settwikilib.cfg will be executed at the beginning of the BEGIN block to make ../lib/CGI.pm and ../lib/CGI/* available.

-- Hideyo Imazu - 2017-06-01

Thanks for this info - it hadn't occurred to me that LocalLib.cfg problems would be an issue. Nevertheless, I think that the whole setlib.cfg + LocalLib.cfg + LocalSite.cfg stuff could use some rework.

• Unfortunately, incorporating the CGI.pm modules into the core would improve nothing. They still would not be available in time. At the time the TWiki scripts are executed, the lib directory which holds both TWiki core and contrib libraries is not in @INC; all we have is bin. That's why setlib.cfg and LocalLib.cfg are located in bin: It is their job to make the TWiki libraries available.
• Of course I agree that problems in LocalLib.cfg should be visible to the sysadmins.
  • However, I'd locate the problem in setlib.cfg which simply throws away any errors in LocalLib.cfg by doing a eval 'require...' and then ignores any return information in $@. This is bad. It is fine to continue if there is no LocalLib.cfg at all, but this is no justification to ignore all other errors as well.
  • The difference between Carp (which is available in all versions of the Perl core) and CGI::Carp (which vanished in Perl 5.22) is that CGI::Carp does some nifty apache-like formatting of the error message. Before we start making the bootstrapping stuff even more complicated by splitting setlib.cfg I'd rather replace CGI::Carp by plain Carp in this early stage of processing. I also think that fatalsToBrowser is no good idea in this stage because problems with LocalLib.cfg will affect each and every request, and there is nothing a TWiki user can do about it.

-- Harald Jörg - 2017-06-01

Back to the drawing board with regard to the CGI::Carp stuff:

• Problems in LocalLib.cfg should be displayed, which is why CGI::Carp::confess or Carp::confess was set to $SIG{__DIE__} at the beginning of the BEGIN block originally.

I still agree. However, after locally reverting the changes in Rev. 30317 and 30319, I checked whether and how errors in LocalSite.cfg are reported by introducing various errors (a syntax error, or a plain die, some warnings) into LocalLib.cfg. As I suspected, neither of the errors is reported. The string eval in setlib.cfg captures fatal errors. $SIG{__DIE__}, which just changes the contents of the error message, never springs into action because there is no death. If LocalSite.cfg itself prints to STDERR with warn, the warnings should in the error log with both the old and new version of the scripts. The difference is that in the current version, they are no longer prepended by the timestamp and location, which is exactly what CGI::Carp provides. But it all started because CGI::Carp isn't available before setlib.cfg has done its job. So we could

1. revert to the previous versions and make CGI::Carp a prerequisite for TWiki. I would prefer not to do this because CGI::Carp comes with CGI, and recent versions of CGI are neither compatible with TWiki nor included in current Perl core.
2. Wrap require CGI::Carp in yet another string eval to see whether it is available, and proceed accordingly. This would clutter the scripts with another hunk of border-case garbage.
3. Keep it as it is and examine in more detail what errors in LocalLib.cfg should be caught. A look at $@ after the string eval would be a very decent start. As far as I know, a "missing" LocalSite.cfg should not be fatal, so some heuristics need to be applied.

-- Harald Jörg - 2017-07-16

CGI::Carp again: Right now I am testing on a machine with Perl 5.24 (i.e. no CGI.pm nor CGI/Carp.pm included), and no system or CPAN version of the CGI distribution. The modules which come with TWiki are the only ones on the machine. As expected, this fails to compile because the TWiki libraries are only available after setlib.cfg and friends have been processed.

So a working solution would be to replace CGI::Carp by a plain Carp before setlib.cfg has been processed, and to replace the $SIG{__DIE__} handler by CGI::Carp::confess only after setlib.cfg has been processed.

By splitting setlib.cfg we could achieve that CGI::Carp gets loaded before "the second part of =setlib.cfg=". However, I would like to avoid yet another configuration item to be documented for installation and upgrade.

-- Harald Jörg - 2017-09-06

I am facing the same problems with Perl 5.22.4. I had to upgrade my Perl because of the recurring error 'Insecure dependency in sysopen while running with -T switch at /usr/share/perl5/CGI/Session/Driver/file.pm line 107.' with Perl-5.18.2 / TWiki-6.0.2) (see Support.SID-02145). The support topic suggested to upgrade Perl, but at the end Perl 5.20 did not seem to solve the problem. This topic, TWikiWithCurrentPerl, however says Perl-5.24 will give problems too, so I tried Perl-5.22-4, but helas....

I suggest to specify which Perl version is working well, which is not in the TWkiki's installation guide.

Since there seems to be no complete solution for the problem above yet, I suggest another option for a patch, found on the internet:

Instead of escaping the left or right brace, how about using its unicode equivalent value? Like so:
while ($rest =~ m/^(.*?)(\\)?\$\x7B([^\x7B\x7D]+)\x7D(.*)$/sg) {
for
while ($rest =~ m/^(.*?)(\\)?\${([^{}]+)}(.*)$/sg) {

-- Emiel Van Riel - 2019-01-17

Hello Emiel,

unfortunately we never were able to track down the Insecure dependency with Perl 5.20, which was part of Debian Jessie and therefore widely distributed.

But as far as I can say, the current TWiki version (6.1.0) covers all Perl versions from 5.10 up to 5.28 - but not 5.20. The Release Notes show fixes for several bug items beginning with "TWiki with current Perl". I have updated this topic's form; I thought I could leave it as a "permanent" item but this was a bad idea because we need to keep track of the progress.

If we missed something, please open a bug report!

IF you are having problems with the CGI / CGI::Carp thing, then you should be able to solve it by installing CGI as provided by your Linux distribution (or from CPAN). This will make CGI::Carp available when it is needed (very early in the startup process, before TWiki's own directories are available).

Your alternate regex would also work, of course. The issue with this change in Perl is that regular expressions with a { are extremely frequent because TWiki variables have the syntax VARNAME{...}, and quite often the regular expressions are created dynamically at runtime. Just finding the places which need to be fixed was the main problem, and maybe we missed some, especially in Plugins.

While I admit that I don't do a lot of TWiki development at the moment, I still am committed to make TWiki run with whatever is the "current Perl version". I happen to regularly update my Perl installations, and I have too many TWiki documents which I don't want to lose. But this covers only the core and the default plugins - there are too many plugins I am not familiar with.

-- Harald Jörg - 2019-01-18