So I completed a few hours of looking into TWiki's performance this weekend...overall I'm pretty impressed with all the work that has already gone into this. I expected to come away with a slew of changes that would really improve things but that wasn't the case. However I did manage to eek out around a 15% improvement for at least the view page (I didn't test these changes across the board since only the 2nd change below is easily testable in this manner.

Basic takeaways:

* If possible we should try to break apart TWiki.pm so that view isn't forced to use modules that it doesn't. My simple test improved the performance of view by almost 10%...of course it broke other bin scripts since I didn't do a real refactoring. If we really refactored we should see 10+% performance improvement.

* By compiling the prefValuesRegexCache ONCE instead of 16+ times during a pageload I saw a 4% improvement in page times for view (similar improvement expected across the board). Note this DOES mean that if the Perl scripts were used to serve MORE than one page any changes to the pref values wouldn't be seen until the process died. Currently I don't believe anyone is using a non-CGI mode where the Perl process stays alive longer...but even if they did the impact would be negligible...

If nothing else we should write a hook that clears this cache on every page since it DOES help per-page request since we only compile once instead of 16 or so times!!!!

159a160,163 >
> my @PrefTermRegexCache;
> my @PrefValuesRegexCache;
>
164a169,188 >
> # Since the pref terms and associated values will not change across
> # multiple page requests we compile the regex expressions once since
> # this function is called a dozen or so times per page request.
> #
> # NOTE: This caching mechanism will need to change if we move to a
> # model where a single Perl process serves more than one page...or
> # if we can detect when preferences change then we can recompile.
>
> if (scalar(@PrefValuesRegexCache) ne (scalar(@prefsKeys))) {
> undef @PrefValuesRegexCache;
> for( $x = 0; $x < @prefsKeys; $x++ ) {
> $term = "\%$prefsKeys[$x]\%";
> $PrefTermRegexCache[$x] = qr/$term/;
>
> my $prefsValue = prvHandlePrefsValue($x);# $prefsValues[$x];
> $PrefValuesRegexCache[$x] = qr/$prefsValue/;
> }
> }
>
166,167c190,196 < $term = "\%$prefsKeys[$x]\%"; < $_[0] =~ s/$term/&prvHandlePrefsValue($x)/ge;

---

> $term = $PrefTermRegexCache[$x];
> # $term = "\%$prefsKeys[$x]\%";
>
> my $prefsValue = $PrefValuesRegexCache[$x];
> # my $prefsValue = prvHandlePrefsValue($x);
>
> $_[0] =~ s/$term/$prefsValue/ge;

* We should reorder the OS check in TWiki.pm to do the tests in the order of OSes that dominate the installed codebase. Further we should add linux as the first test since most TWiki systems are probably running under linux:

if ($OS=~/linux/i) { $OS = 'UNIX';

This made a very slight improvement in load time...

* $securityFilter in TWiki.cfg should be pre-compiled since this is used several times per page request:

$securityFilter = qr/.../;

* It would be nice if single quotes were used anywhere variable expansion isn't necessary...but this won't likely improve much.

---

And here are some notes from my testing session....

% strace view

---

stat64("/usr/lib/perl5/5.6.1/Exporter.pmc", 0xbfffeb80) = -1 ENOENT (No such fil e or directory) open("/usr/lib/perl5/5.6.1/Exporter.pm", O_RDONLY) = 5

stat64("/usr/lib/perl5/5.6.1/Carp.pmc", 0xbfffeb80) = -1 ENOENT (No such file or directory) open("/usr/lib/perl5/5.6.1/Carp.pm", O_RDONLY) = 5

---

* Why aren't standard packages compiled??? Ahhhh...currently can't compile any module that requires another module (use is ok) OR uses any modules which requires another module.

See... http://archive.develooper.com/module-authors@perl.org/msg00141.html http://www.perldoc.com/perl5.6.1/lib/B/Bytecode.html http://faqchest.dynhost.com/prgm/perlu-l/perl-01/perl-0104/perl-010403/perl01042918_26350.html

* Appears that Prefs.pm can be compiled...actually all TWiki::* can be cmopiled except Store... Tried this but didn't work...couldn't load Perl module with error:

syntax error at ../lib/TWiki/Search.pm line 1, near "linux" Compilation failed in require at ../lib/TWiki.pm line 106. BEGIN failed--compilation aborted at ../lib/TWiki.pm line 106. Compilation failed in require at ./view line 25. BEGIN failed--compilation aborted at ./view line 25.

---

# dprofpp -O 10 Total Elapsed Time = 1.343305 Seconds User+System Time = 1.314872 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 28.1 0.370 0.449 16 0.0231 0.0281 TWiki::BEGIN 19.0 0.250 0.858 6 0.0416 0.1431 main::BEGIN 11.4 0.150 0.150 14 0.0107 0.0107 CGI::_compile 3.04 0.040 0.090 4 0.0100 0.0224 CGI::BEGIN

Strange...for a single run of view why did we: - load TWiki::BEGIN 16 times... - load CGI::_compile 14 times...

I modified TWiki.pm briefly for the view page and removed the following use statements as a test.

use TWiki::Search; use TWiki::Form; use TWiki::Net;

And the profile changed quite a bit...

% dprofpp -O 10 Total Elapsed Time = 1.182289 Seconds User+System Time = 1.152289 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 23.4 0.270 0.309 13 0.0208 0.0238 TWiki::BEGIN 21.7 0.250 0.708 6 0.0416 0.1181 main::BEGIN 11.2 0.130 0.130 14 0.0093 0.0093 CGI::_compile 6.86 0.079 0.132 6 0.0131 0.0220 TWiki::getRenderedVersion 2.60 0.030 0.030 4 0.0075 0.0075 TWiki::Plugins::BEGIN 2.60 0.030 0.090 4 0.0075 0.0224 CGI::BEGIN

Removed THREE uses of TWiki (and therefore calling of TWiki::BEGIN)...cut at least .1 seconds of System/User time from the total run of 1.2 seconds...WOW...almost a 10% improvement for free!!!

Hmmm, is BEGIN called once for each use statement? Appears so...I added a "use Carp" and another BEGIN showed up...I thought there was one BEGIN per package that got executed and called require/import on all the used packages...which would trigger BEGIN calls on all those other packages...strange.

I noticed there are two "use lib ..." calls in view...the docs for lib.pm shows that you can pass in a list of directories instead of making separate calls.

use lib ( '.' ); use lib ( '../lib' );

vs

use lib ( '.', '../lib' );

Appeared to make NO measurable difference...in DProf...running Benchmark also yielded nothing tangible. Good to know.

At least it does one less execution of main::BEGIN...so why not...

Hmmm, interesting test...so I added the following to the top of TWiki.pm:

for (my $i = 0; $i < 10000000 ; $i++) { $i++; }

And reprofiled:

Total Elapsed Time = 8.9354 Seconds User+System Time = 8.8954 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 90.3 8.040 8.459 6 1.3400 1.4098 main::BEGIN 2.92 0.260 0.309 13 0.0200 0.0238 TWiki::BEGIN

So it appears that any code executed when loading a module (i.e. file scoped initialization) actually is counted in the time for the file that is DOING the loading!!!

That means main::BEGIN includes all the initialization time for the modules that it is using...so we should do a quick pass to see what is actually initialized on every module load and if there is a way to do some lazy initialization.

I see that we have a do "TWiki.cfg" here...nothing too bad here. Would have been nice if we used ' instead of " so that Perl didn't have to scan all the quoted strings.

* Hmmm...interesting...reordering the regexps to determine the OS in the order of most frequent to least frequent AND adding linux as the first seemed to slightly improve the time (-0.0008 seconds for TWiki::BEGIN on average).

if ($OS=~/linux/i) { $OS = 'UNIX';

* attachAsciiPath...this is a regexp...why aren't we pre-compiling it?

Ahhh, if we want this (see below note) then it should be:

$attachAsciiPath = qr/\.(txt|html|xml|pl)\$/;

I guess for single process execution this is worse because you are ALWAYS compiling it even if you don't need to. But if we switch to the model where a single Perl process may serve multiple requests we should fix this.

* Same with $securityFilter, $uploadFilter...except security filter might work out better since it's used multiple times even on a single page request...so we SHOULD at least change securityFilter to:

$securityFilter = qr/.../;

-- RyanSnodgrass - 03 Mar 2002

It would be worth considering mod_perl (ModPerl) and SpeedyCGI here - I got a 10 times speedup just from implementing mod_perl, and SpeedyCGI can be used even for TWikiOnWebHostingSites. There are some issues getting mod_perl working with TWiki (I couldn't get Apache for Windows + ActivePerl + Cygwin RCS to work), so it would also be good to look at this in the quest for performance.

Another interesting area is BrowserAndProxyCacheControl - as well as fixing some data-loss bugs with IE5, this could help scale larger sites and improve response times for view (many intranets and ISPs use proxy caches, some transparently to their users). If controlled properly, the site administrator could make the trade-off between timeliness of pages and load on the web server - currently the trade-off is quite arbitrary, with some proxy caches imposing caching without regard to what's reasonable for TWiki, but fortunately most proxy caches do respect the cache-control headers of HTTP.

On a somewhat related note, I'm also rewriting the statistics script so that it will scale to large systems without consuming a lot of memory and getting killed as a result (on sites that limit this) - see StatisticsUsingLessResources.

-- RichardDonkin - 03 Mar 2002

Yes, doing testing under mod_perl was my next step since then you can really narrow in on the performance outside of one-time initialization costs. I didn't realize it was so easy to get up and running under mod_perl though!

One of my suggestions above works only in standalone CGI mode currently (the cached pref values) but this can easily be made to do the right thing across requests. I'll play around with actually creating a patch that does the right thing for all cases.

-- RyanSnodgrass - 04 Mar 2002

Yes, I would advise that you use mod_perl too, as some of your optims are irrelevant with mod_perl (breaking TWiki.pm in small pieces...)

-- ColasNahaboo - 05 Mar 2002

CategoryPerformance