Core code benchmarks for TWiki. Unzip the attached zip in a neutral directory.

• Requires Storable (CPAN)
• Requires a local installation of Athens
  • Must be located at URL $server/athens/ and physical directory $installdir/athens
  • AthensRelease download : http://twiki.org/p/pub/Codev/Release/TWiki20011201.zip
• Requires the installations to be benchmarked to be at $server/version/ and $installdir/version, where version is the local name for that version e.g. "beta20040816" or "beijing"
• Each installation has to have a 'benchmark' script installed in it's bin dir. This script is derived from the view script, and is therefore different for each major release. Appropriate benchmark scripts are at the top level of this zip; copy the right one to the bin directory of the installation and rename it 'benchmark'.
• Requires a test page in the same place on all the installations; for standard benchmarks this should be WhatIsWikiWiki.

For examining performance of other functions of TWiki, then pick an appropriate page that exercises those aspects as much as possible. It is wise to stick to pages in TWiki web as far as possible, as they tend to have been around for a long time.

The benchmarks are run using the experiment script. This script runs the benchmark on Athens first (to normalise AthensMarks) and then repeats the experiment on a list of other installations. Edit the script as appropriate for the installations you want to benchmark.

Benchmark results are reported in a table that shows the plugins installed in the installation, together with an absolute number indicating the time taken to render the page, and the performance of the instalation in AthensMarks. AthensMarks are defined as the percentage of athens performance delivered by the installation; so 100 AthensMarks is the same performance as athens; 33 AthensMarks indicates performance 3 times worse than athens; 300 AthensMarks indicates performance 3 times better than athens.

The benchmarks are for the core code only i.e. the overhead of client/server comms is not taken into account. This means that factors such as time taken to download CSS or Javascript files are not being measured. For that, use ab based benchmarks.

Latest core benchmarks

Run by CrawfordCurrie - 24th October 2004

Release	Skin	Time per page	AthensMarks
athens		.3266	100.0000
beijing		.3900	83.7400
cairo	classic	.4783	68.2800
MAIN	classic	.5450	59.9200
DEVELOP	classic	.4233	77.1500

Contributors: CrawfordCurrie, RafaelAlvarez, FrankHartmann

What happened on MAIN above? Nobody should be checking in any code that decreases performance by that much. The performance deterioration in Cairo is alrady hard to swallow, heaping even more on is really not good. Can we back these changes out or move the DEVELOP branch over?

-- ThomasWeigert - 25 Oct 2004

I honestly don't know; there have been no significant code changes. Here are some possible reasons, all of which probably contribute.

1. a copy-paste of code for internationalisation that resulted in a second "do TWiki.cfg" in Render.pm.
2. extractParameters
3. increased code volume
4. an artifact of perl, showing up in the order the benchmarks are run
5. changes to the templates, resulting in greater processing requirement

DProf doesn't indicate any specifics. Try doing a svn diff -r 1742 in MAIN, and see if you can spot the guitly party.

Note that even after repeated runs, this result is consistent; MAIN is definitely 7-8 AthensMarks slower than Cairo.

IMHO we should not merge DEVELOP until a few more beady eyes have reviewed it. It may be faster because it isn't doing something.

-- CrawfordCurrie - 25 Oct 2004

Good point about the I18N cut-and-paste - the do "TWiki.cfg" in Render.pm is completely unnecessary, and the BEGIN block should look more like Search.pm, i.e.:

BEGIN {
    # Do a dynamic 'use locale' for this module
    if( $TWiki::useLocale ) {
        require locale;
        import locale ();
    }
}

The cost of this code should be quite low since it is just doing a use locale and the locale.pm module has already been require'd by TWiki.pm.

Now committed on MAIN in SVN.

-- RichardDonkin - 25 Oct 2004

I think I found a "bug" in the experiment.pl script. In the run sub, see the following code:

    my ( $code, $skin ) = @_;
    if ( $skin ) {
        $skin="?skin=$2";
    }

$2 is not defined in the context, so $skin will get the value ?skin=. This causes that most cairo-based distros will be measured using the pattern skin.

it should read:

    my ( $code, $skin ) = @_;
    if ( $skin ) {
        $skin="?skin=$skin";
    }

What's the difference?

With the original code these are the results (DEVELOP and trunk versions are 3183 and 1819):

TWiki core code benchmarks

Release	Skin	Plugins	Time per page	AthensMarks
base/athens		DefaultPlugin, InterwikiPlugin	0.561111111111111	100
base/beijing		DefaultPlugin, InterwikiPlugin	0.63125	88.8888888888889
base/cairo	classic	DefaultPlugin, InterwikiPlugin	1.036	54.1613041613042
base/trunk	classic	DefaultPlugin, InterwikiPlugin	0.923333333333333	60.7701564380265
base/DEVELOP	classic	DefaultPlugin, InterwikiPlugin	0.74	75.8258258258258

with the fix:

TWiki core code benchmarks

Release	Skin	Plugins	Time per page	AthensMarks
base/athens		DefaultPlugin, InterwikiPlugin	0.56	100
base/beijing		DefaultPlugin, InterwikiPlugin	0.622222222222222	90
base/cairo	classic	DefaultPlugin, InterwikiPlugin	0.951666666666667	58.8441330998249
base/trunk	classic	DefaultPlugin, InterwikiPlugin	0.926666666666667	60.431654676259
base/DEVELOP	classic	DefaultPlugin, InterwikiPlugin	0.715714285714286	78.2435129740519

-- RafaelAlvarez - 08 Nov 2004

Ach, sorry, Frank pointed that out before and my local version is correct already, it's just the uploaded package that was wrong. I have refreshed it.

-- CrawfordCurrie - 09 Nov 2004

Note: AthensMarks have been discredited/ignored by various authors and as a result I have stopped maintaining the scripts. Unfortunately no-one has come up with anything better. The TWiki::Monitor class offers a way to do detailed performance analysis, but there is no agreed TWiki benchmarking approach that can give a single simple number.

-- CrawfordCurrie - 28 Oct 2007

CategoryPerformance