PersistentPerl - Speed up TWiki Perl scripts by running them persistently

Summary: TWiki typically runs under CGI, which can be quite slow - PersistentPerl, also known as SpeedyCGI, is one way of speeding up TWiki, by compiling and loading its Perl scripts into memory just once, avoiding this overhead on every TWiki page.

This is used by TWikiVMDebianStable to provide a fast and easy to install version of TWiki on Windows.

PersistentPerl is a more recent name for the SpeedyCGI tool. It's an alternative to ModPerl for Apache2. I had to hack the build files for it to build properly under Debian (GNU/Linux) but it seems to work fine.

All I had to do after installation was create add the mods-available/perlperl.{load,conf} files, run the rewritehashbang.pl script in tools/ and force-reload the daemon. Runs smoothly, and quickly.

Oops silly me forgot the URI: http://daemoninc.com/PersistentPerl/.

-- EricCote - 25 Feb 2006

Hmm, seems it's too persistent. I click on a link to go to another web, and I see the same topic before the click. I guess TWiki doesn't support this yet. I'll see if I can hack something into place.

Adding in PerlSendHeader On helps a lot . This is what I did to bin/.htaccess on the suggestion from above:

PerlSendHeader On
SetHandler persistentperl-script
Timeout 600
Group SRWiki

Doesn't help for very long, alas. Looking into it after dinner.

-- EricCote - 25 Feb 2006

OK. Disabled server-level Group setting, and set it per vhost; disabled the AddHandler; set Timeout to 60; and removed 'per' from the hash bang in natlogon, login, logon otherwise logins didn't work. So far edit seems to function normally PerlPerlised.

-- EricCote - 25 Feb 2006

-- Contributors: EricCote

Discussion

I'm curious to know how this compares with SpeedyCGI and ModPerl, in terms of ease of setup, compatibility with TWiki, required dependencies such as being Apache2 administrator, and so on (once you have it fully sorted out). I would generally expect that the view script and a few others should run with PersistentPerl, as discussed in SelectiveModPerl.

-- RichardDonkin - 25 Feb 2006

PersistentPerl is SpeedyCGI's new (well, at least 2 or 3 years old new) name

-- WillNorris - 27 Feb 2006

What kind of performance improvement are you experiencing? I couldn't get it to really work except when I change the first line for bin/view to lauch perperl instead of perl. With that I am getting like 10% faster... Is it worth the risk?

-- JohnnyKwei - 04 Mar 2006

Eric, could you refactor this topic so that it is a concise "how-to" document? I already linked it from PerformanceSupplement.

-- PeterThoeny - 13 Mar 2006

Updated to reflect fact that PersistentPerl = SpeedyCGI.

-- RichardDonkin - 16 Mar 2006

Peter Keller gave me the following advice, for which I am most grateful:

Premature end of script headers can be hard to diagnose. I've tried 'lynx -trace' on your URL, but I can't see anything during the http negotiation that looks different from a normal CGI request, so it must be the server-side setup. A few ideas:

Double-check that you have got the #! line right in twiki/bin/view

Does everything work OK if you use normal perl instead of perperl?

Are you running the command line prompt as the same user that runs the httpd process?

Have you checked the environment? Run a CGI script that contains /usr/bin/env or printenv, and check that it is sane. Set up an identical environment in an interactive shell, and see if your script still works with persistent perl. If not, try 'ldd -r perperl' and see if shared libs cannot be found.

Kill any running perperl/perperl_backend processes and delete any /tmp/perperl.* files. When you query the URL again, do new perperl processes start? Do any new temporary files appear in /tmp?

Is your /tmp directory OK? (Persistent perl depends on this). Is the user running the httpd process allowed to write to it? Is it full? Is TMPDIR set to some unusual value in the server-side environment?

-- MartinCleaver - 23 Mar 2006

Just would like to add that it seems PersistentPerl can't cope with high load - I get "temp file is corrupt" errors on our increasingly-popular twiki server. Probably a race condition with multiple accesses?

-- WoutMertens - 23 Apr 2008

This might depend on the plugins that come with your twiki. Maybe some of them create temp files in a bad way?

I just recently had serious problems with speedy-cgi and Image::Magick blocking all of the apache process. (Image::Magick is used by some popular image plugins) That was a known error in Image::Magick which does not occur with Graphics::Magick a more recent fork of Image::Magick...

-- MichaelDaum - 24 Apr 2008

You don't even need to load the apache module for speedy_cgi -- just edit the #! line in your scripts to be /usr/local/bin/speedy instead of /usr/bin/perl. I did it for just the edit and view scripts. The rest are so seldom called I don't want them in memory. The first page load will take a small hit in time, but after that the memory usage and speed are very good.

-- VivekKhera - 15 Sep 2008