I have a web of some 48,000 topics, over which I am trying to do a regular expression search. I have discovered that 4.0.5 runs about 7-10 times slower than Cairo for the same search. The curious thing is that I can't explain why.

Here's the setup:

1. data and pub directories somewhere
2. cairo install which sets dataDir to point to them
3. 4.0.5 install that sets dataDir to point to the same place

Here's what I tried:

1. Verified that they are both searching the same number of topics, with the same string (they are)
2. Verified that they return the same search hits (they do)
3. Verified that they are batching up the greps into 512 topic blocks the same way (they are)
4. Verified that the command being used is identical (it is)
5. Used Benchmark qw(:hireswallclock) to isolate the badly performing code down to the $sandbox->sysCommand call in RcsFile::searchInWebContent.
   • This call accounts for 95% of the runtime (the rest of the runtime is roughly equivalent in Cairo and 4.0.5, so can be ignored).

So far, so good. The code is searchInWebContent is almost unchanged since Cairo (it was cut-pasted from Search.pm), except it now uses the Sandbox. So the problem must be in the Sandbox, right? So I tried recoding the sysCommand call to use backticks directly, instead of via the sandbox. This is where it gets wierd. This is actually fractionally slower than using the equivalent Sandbox call!

While iterating on this I made the search code (in RcsFile.pm) look more and more like the Cairo code from Search.pm, until it was almost identical. But the search in 4.0.5 is still far slower than Cairo! When I analyse the actual backtick execution times (using Benchmark), the call within the 4.0.5 codebase is observably 10 times slower than the identical call within the Cairo codebase! (/bin/egrep -i -l -- searchstring file file...). A typical Cairo grep is 0.0235548 wallclock seconds, but the identical grep in the Dakar code is 0.24 wallclock seconds!

Any ideas? Can anyone else reproduce this? Could this be to do with some magic perl variable e.g. setting something to use buffered IO?

---

Later: I just realised that 4.0.5 was running under mod_perl, so in a spirit of canonicalisation I turned off mod_perl and bingo! Suddenly the performance of the searches rocketed up to the Cairo level (in fact it is now slightly faster).

I'm still looking for an explanation, though, because mod_perl is important to acceleration. Do we have to stop spawning subprocesses to do searches from mod_perl?

See http://stason.org/articles/perl/mod_perl-2002.13.html

This may explain why people who have tried mod_perl have reported no speedup. It may be because they are looking at pages containing searches.

I am investigating PurePerlBackendForSearch.

---

Later again: I swapped in Rafael's PurePerlBackendForSearch, and performance is about 4X worse than forked grep without mod_perl, and therefore around twice as fast as forked grep with mod_perl.

Any other ideas would be most welcome!

I raised Bugs:Item3443 for further tracking.

-- Contributors: CrawfordCurrie - 12 Jan 2007

Discussion

Good to know that search is actually slower with mod_perl. What with FastCGI? TWiki.org is running on FastCGI right? I never used FastCGI I should be reading a bit about that. Anyway it's also nice to find out about things like DBCacheContrib. Is there a place/topic where senior TWikier list the extensions they find indispensable, useful or just kool?

• YetAnotherDBCacheContrib attempts to be as close to TWikiSearch as possible. It is definitely faster than the grep based search, in particular when the search is complex. However, you do incur a hit in loading the cache upon topic read. -- TW

-- StephaneLenclud - 13 Jan 2007

CC, I may be confused, but reading the bottom of the article you cite it appears that the overhead on fork does not occur for system calls. Am I misunderstanding that article? The gist seems to be that there is a huge overhead when writing to the memory of the apache process, but forking a sys call would not? Or is it that when we pass the results back from the sys call that it writes into our memory, triggering copying the process state?

I did not see the code for PurePerlBackendForSearch in that topic. Can you point me to that?

-- ThomasWeigert - 13 Jan 2007

That section of the article deals with the overhead from system(), not the overhead from fork().

When I was experimenting, I tried disabling the safe pipes (which meant the Sandbox code fell back to a backticks solution) and it was just as slow. I concluded that the backticks also spawn a subprocess and incur the same overhead, despite mapping directly to a system() call with no shell metacharacters.

Rafael's code is attached to PurePerlBackendForSearch. I attached a patch to this topic for a much simpler implementation, which kicks in if mod_perl is used.

-- CrawfordCurrie - 18 Jan 2007

Rather than hardcoding this to mod_perl I think it is better to use a configure flag. There are other env that can benefit from a pure Perl TWiki (assuming performance is OK), such as TWikiForWindowsPersonal.

-- PeterThoeny - 18 Jan 2007

If a search with twiki+mod_perl is slower because of an increased fork overhead (the apache process being coppied), then mod_perl is out of the window. I expect speedy-cgi to perform a search much easier than mod_perl as speedy does not reside in the apache process space but in a separate perl interpreter with twiki in it and no more. So a fork() in there does not incur the same overhead as mod_perl and should be as fast/slow as in a non-accelerated perl setup.

-- MichaelDaum - 19 Jan 2007

Peter, good point, I considered a configure flag but I want to try to reduce the number of configure options, not increase them. configure is already far too complex. Someone needs to do the UI work on configure required to hide the EXPERT options.

Michael, correct. Speedy does not suffer anything like as badly as mod_perl. However mod_perl is common in hosted and corporate installs, while speedy is not. If we can find a solution to search that doesn't die with mod_perl then we should use it. Someone needs to write a grep in a shareable library for Linux, so it can be called from TWiki using XS without spawninbg a subprocess. As shown by my patch, it doesn't need to be very clever, though it would require an RE library.

-- CrawfordCurrie - 19 Jan 2007

Would it not be possible to do SelectiveModPerl so that the search script is run via CGI, avoiding this problem, while retaining ModPerl for most short-running scripts such as view, edit, etc?

BTW I wouldn't agree that ModPerl is common in web hosting installations - it's very hard to find a ModPerl hosting company at all, unless you pay a fair bit extra for a virtual private server (VPS, i.e. root access to a virtual machine on shared server) or a lot extra for dedicated servers. Some hosts, such as Dreamhost (see DreamhostSetupNotes), do have FastCGI support but I don't believe TWiki supports that as standard.

• Richard, what is it about FastCGI that does not let it work with TWiki? -- TW
• Skimming the dreamhost doco it appears that FastCGI works by splitting scripts into a section which executes once and sections that execute every time the script is invoked. Thus it appears that we would have to divide each TWiki script into the piece that loads (uses, requires) all the code for TWiki and the plugins, and maybe does some initialization, and the normal code in the script. This may in fact be an interesting idea, but sure would cause a maintenance problem? -- TW

-- RichardDonkin - 20 Jan 2007

I sketched out a bourne shell script which takes the "Command" and "Temp filename" as the arguments, then reads from STDIN, and runs the command on each input line.

All TWiki has to do is run the script through a pipe, split the topic list into 512 name chunks writing into the pipe. The output is appended to the temp file, which TWiki reads in and parses as usual.

This should result in only a single fork (for the pipe), no matter how many topics are being searched. I haven't completed the test, I am hoping this is faster than the pure perl implementation.

BTW, rootr.net has mod_perl for $25/month.

-- CraigMeyer - 20 Jan 2007

Richard, no, the problem is endemic, wherever there is a %SEARCH.

Nice one, Craig. However there is a even more efficient solution, both in terms of code and of execution time. As I suggested above I have written an XS module that has a simple grep and is linked to TWiki. It runs faster than the current implementation in both mod_perl and outside it. I'm going to package it as a contrib so people can choose if they want to install it or not.

-- CrawfordCurrie - 20 Jan 2007

The attached diff has my fast search code (diff against latest rev, though it should also work with 4.1 and 4.0.5) as checked into subversion.

Apply the diff, and

$ cd tools/native_search
$ perl Makefile.PL
$ make install

You will probably need to be root. If anyone can help to package this so you don't have to be root, please do so.

-- CrawfordCurrie - 21 Jan 2007

This is very nice work, CC - impressive progress on this issue.

-- SteffenPoulsen - 21 Jan 2007

Nice that we have a fix for that I wanted to post something on the defect report Item3443 but it does not work right now (yet another twiki.og problem? )!?!

• I can't find a link to the documentation
• No date stamps associated to people comments on Item3443. I find that rather confusing

-- StephaneLenclud - 27 Feb 2007