Tags:
create new tag
, view all tags

TWikiPerformanceEnhancements

I've been doing a lot of plugin development for our TWiki site, and discovered a few areas I'd like to improve upon in the core code. Feel free to add more stuff here if you want. I'll start with the Prefs.pm, since that's where I saw the most execution time spent.

Prefs.pm

I rely heavily on preferences for implementing my plugins, and I noticed the more a page used the plugin, the slower it ran. I decided to turn on code coverage and performance metrics in the Perl executable I was using (-dDProf and -dCoverage). Turns out there were large amounts of unecessary time spent looping in the Prefs.pm module.

I noticed that Prefs.pm was using arrays to implement a hashing function, which is already built into perl, and much faster than this particular implementation. I've attached an updated Prefs.pm that works with the September release. Original code is commented out and replaced with the same code that works based on perl hashes.

If you have any plugins that use preferences heavily, you'll notice a decent performance increase.

I still have some work to do to further improve performance in that module, but that's what I've done so far.

Skin Performance

I've also noticed a few things to watch out for when implementing fancy skins to improve download performance, without sacrificing too much look & feel. I'll be posting a new Theme-based skin plugin that provides a way to customize the way a skin looks based on some settings and attachments in a theme topic.

Some things to watch out for:

  • Nesting tables just to get a border takes quite a long time to render in netscape, compared to IE or Opera. Wait until your user base is no longer using Netscape 4.x to add borders, and do it with style sheets.
  • Huge menus that get constructed, only to be selectively hidden by the browser, based on style-sheet settings. Things that are designed to be hidden from view, especially if there's a lot of it, end up taxing the browser's rendering engine. If pulldown menus are required, that's fine, but it's not worth the download and render time if the items are never displayed on the page. If it's not going to be displayed, don't put it on the page.

I'll think of more stuff eventually... If anyone knows of other items that cause slow rendering/processing/download times, please add it.

Core/Plugin code performance tweeks.

Most of this stuff can be found in the O'Reilly Perl collection, but I'll mention it here, along with the TWiki-specific stuff.

  • Multiple calls to the same plugin's rendering subroutines can be expensive, especially if the plugin wasn't efficiently implemented. Move any initialization from rendering routines, such as grabbing of preference information based on web and topic name into the plugin's initialization subroutine.
  • Calling subroutines in regular expressions that get used frequently is expensive. Try to use elaborate replacement patterns instead, to prevent unneccesary stack manipulations.
  • Regular expressions that can't be compiled once and run many times are expensive. Every time the expression is evaluated, it will be recompiled, unless the "o" option is added at the end of the expression.
  • Use hashes instead of arrays indexed by other arrays.
  • Eliminate unnecessary loops by using hashes.
  • Use "foreach" loops that exit once the required value is detected instead of grep.

I'll probably be going thru the codebase and running DProf and Coverage on it to look for more areas that could be improved.

-- PeterNixon - 18 Dec 2001


Gotchas with compiled regexps and mod_perl

I am using mod_perl for twiki execution. So you need to be careful about the regexp compilation optimizations. There is more about this in the mod_perl documentation. Basically the crux is this: When running without mod_perl, a new invocation of twiki results in a new recompilation since the compilation of the variable/pattern hasn't been done. In a mod_perl setup, it is possible to get a call into a precompiled regexp where the actual value of the regexp is different, but the compiled (older) version of the regexp is still used.

Is anybody involved in cleaning up TWiki for mod_perl execution?

From the mod_perl_traps.pod doc:

COMPILED REGULAR EXPRESSIONS

When using a regular expression that contains an interpolated Perl variable, if it is known that the variable (or variables) will not vary during the execution of the program, a standard optimization technique consists of adding the o modifier to the regexp pattern, to direct the compiler to build the internal table once, for the entire lifetime of the script, rather than every time the pattern is executed. Consider:

        my $pat = '^foo$'; # likely to be input from an HTML form field
        foreach( @list ) {
                print if /$pat/o;
        }
This is usually a big win in loops over lists, or when using grep or map.

In long-lived mod_perl scripts, however, this can pose a problem if the variable changes according to the invocation. The first invocation of a fresh httpd child will compile the table and perform the search correctly, however, all subsequent uses by the httpd child will continue to match the original pattern, regardless of the current contents of the Perl variables the pattern is dependent on. Your script will appear broken.

There are two solutions to this problem.

The first is to use eval q//, to force the code to be evaluated each time. Just make sure that the eval block covers the entire loop of processing, and not just the pattern match itself.

The above code fragment would be rewritten as:

        my $pat = '^foo$';
        eval q{
                foreach( @list ) {
                        print if /$pat/o;
                }
        }
Just saying
        eval q{ print if /$pat/o; };
is going to be a horribly expensive proposition.

... there is some more, but this is the basic warning.

-- JohnRouillard - 18 Dec 2001

Excellent point... This is also true for installations using SpeedyCGI.

-- PeterNixon - 18 Dec 2001

I have run under ModPerl for some time and alter a few regexps to be compatible. You'll see a few comments stating why no optimization because of ModPerl.

-- JohnTalintyre - 19 Dec 2001


CategoryPerformance
ChangeProposalForm
TopicClassification FeatureRequest
TopicSummary speed up performance, especially plugins, by implementing hash'd Prefs.pm (attached)
CurrentState UnderInvestigation
OutstandingIssues there is likely a significant amount of code drift since this patch was developed.
RelatedTopics

InterestedParties

ProposedFor

TWikiContributors PeterNixon
Topic attachments
I Attachment History Action Size Date Who Comment
Perl source code filepm Prefs.pm r2 r1 manage 7.7 K 2001-12-18 - 15:54 PeterNixon Modified to implement Prefs storage in a hash
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r8 - 2006-04-25 - CrawfordCurrie
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.