Question

Guys, is it possible to preload the scripts in the "bin" folder along with the TWiki modules in the "lib" folder?

I'm running mod_perl on apache 2.2.6 with perl 5.8.8, fedora core 8.

Environment

-- TWikiGuest - 14 Feb 2008

Answer

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.

It may be possible, but it isn't worth the effort. The scripts in the bin folder are those which actually trigger execution. These scripts are really short, their compilation time should be neglegible, and their file's timestamp is evaluated by mod_perl to check whether a re-compilation is needed.

You could force preloading of the scripts by explicitly requiring them in the startup script, but this needs care to capture the output which would otherwise clobber your error log. Some scripts like bin/statistics might slow down the startup considerably, without any real value. So don't do that.

I'd say it is fully sufficient to preload the modules in lib in the mod_perl startup. Don't just preload those which are directly referenced by the scripts, many modules are (for speeding up non-persistant interpreters) compiled during runtime, which in mod_perl means once per process, and not in shared memory.

A simple profiling trick is to have a small additional mod_perl script somewhere on the server which inspects and prints the list of actually accumulated modules in %INC.

I am running such a mod_perled TWiki in a VM with 256MB RAM, two dozen plugins, serving some 20000 requests per day without using swap.

-- HaraldJoerg - 14 Feb 2008

Harald, thanks for this response :). I actually tried preloading the lib folder modules. But then apache wouldn't start. It kept generating lots of errors complaining that the libraries could not be found. Should we let setlib.cfg to preload so that it sets all the environment variables properly? I'm slightly lost in that area. Btw, 20000 requests a day is pretty awesome!

On my server with 1GB of RAM and 2.2Ghz CPU, during load testing, it is not able to serve more than 150 requests at a time. The requests are sent from different systems in parallel. Though, during actual usage, there would not be so much concurrent requests, it would be better to get a robust system in place so that, even if we have a dramatic increase in number of users, the system responds.

For loading one of the Twiki libs, I did,

unshift @INC, '.';
require '/var/www/html/twiki/bin/setlib.cfg';
require TWiki::UI::View;

After giving the above in my mod_perl startup script, httpd doesn't start. Please let me know where I'm wrong. Thinking of how to preload these libraries...

-- TWikiGuest - 15 Feb 2008

I wouldn't bother with setlib.cfg in a mod_perl startup script, but simply define explicitly where your modules are. That scary unshift @INC, '.'; is just some sort of kludge to make TWiki location independent. You know where your modules are in your installation, don't you? So you'd just write:

# Set library paths:
#    * CPAN libraries from the TWiki distribution
#      (you might want to install them in your system wide
#      Perl installation)
#    * TWiki's own libraries
use lib '/var/www/html/twiki/lib/CPAN/lib';
use lib '/var/www/html/twiki/lib';

# Explicitly compile all of CGI
use CGI();
CGI->compile();

use TWiki::UI::View;
use TWiki::UI::Edit;
# ...continue as you like, especially take care for stuff for which
#    compilation is deferred.  Needs some profiling to get a perfect
#    list for your installation.

1;

...But by the way, don't overestimate the 20000 hits per day. This is server load, true, but not actual usage by TWikizens. I have only few simultaneous requests since most of the requests are from piloting a greedy intranet search spider with only five to six simultaneous requests, coming in intervals but distributed on 24 hours a day.

Good luck!

-- HaraldJoerg - 15 Feb 2008

Whoa! Harald, that's awesome! Will surely try this and let you know. Thanks a lot for the help.

-- TWikiGuest - 18 Feb 2008