SID-00446: How to memcache TWiki?

Status:	Answered	TWiki version:	4.3.0	Perl version:	5.10
Category:	CategoryPerformance	Server OS:	Debian / Ubuntu Server 8.10	Last update:	16 years ago

Hi, our twiki at work is primarily an application for following on project status, reports, calendars, etc...

The system's base architecture is: Main topics are project based and are composed of template includes (ie: header, middle, footer) that have tables with searches for the main topic's children information. In the Main topics you can create new children topics at will, and their information will appear on the main topic which is basically just a fancy project status report. Most of the "real" information is entered with webforms (for ease of use) on the children topics and it can include dates, so a search on a calendar page puts the events or deadlines on the calendar itself, etc...

So... as you can see by now, the problem is that 90% of the twiki is based on searches. and they can't really be cached for a long time or the information on all the status/calendar pages is outdated.

All the searches are starting to slow down the system, and I can forsee a problem in the future.

This is a small installation, but after 6 months we're touching 1200 or so topics. They're really small in size (includes for webforms, and formfield data) so the entire twiki folder isn't very big < 100MB.

I've read of memcached and think it would be a solution for speed, having the view script access data from the memcached daemon, and the save script update the cache on that particular topic. but if this is possible, is there a how-to setup a memcached twiki anywhere??? and would this information be topic info (affecting searches before rendering) or would it be cacheing pre-rendered pages?

Or... is there another way to pre-load all the topics data into memory, so the searches are faster? As It would take years of data before we have to buy more Ram to hold it in.

Or is the DBCacheContrib approach an easier and better solution to this problem?

Thanks in advance.

-- HugoDeLaCerda - 2009-07-30

Discussion and Answer

Yesterday I tested the DBCache solution on a clone system, and the performance gains are not really noticeable at this point, and the formatting of the information gathered by the DBQueries is not as configurable as the searches'.

Having a mixed search and DBCache system doesn't make much sense as a solution either, since the performance gains are minimal if we still need to use some searches because of formatting, and the save topic time get's slower because of DBCacheContrib updating the caches we're only partially using.

That's why I changed the topic name to "How to memcache TWiki? "

I believe this is something a lot of people out there would like to know how to do.

Please help

-- HugoDeLaCerda - 2009-07-31

You can improve the speed if the pages you are searching for have a common base name. For example, if you have pages such as SID1234, SID1235, etc, you can apply a topic="SID*" parameter to your search. This will improve performance.

Look also into the VarCachePlugin. It can cache pages with slow searches.

Longer term, TWiki should be enhanced with a database backend that indexes form data for faster queries.

-- PeterThoeny - 2009-08-14

Thanks for the suggestion Peter, however the performance difference was not noticeable.

I read somewhere about TwikiCache::Memcached or something like that.

This approach as I understand would significantly improve on performance as I now believe the searches are not the big culprit anymore as much as it's the multiple plugins+searches+page render times that add up and a memcached approach would really make a difference.

So If anyone knows how to implement this and would like to share it, I'm sure it would benefit many of us.

-- HugoDeLaCerda - 2009-08-14

Hi HugoDeLaCerda,

To my knowledge - currently we do not know how to use memchace with TWiki. It must be very easy to integrate memcache with the applications where data is stored into the databases. With TWiki, all data is in standard text files.

We really need to have TWiki::Cache::Memcahe module which helps us to improve

• Rendering of the topics
• Meta-data retrieval
• Search (TWikiForm field based searches)

I am ready to work on this/share ideas/put some time on this to collaborate-let me (sopan.shewale at gmail.com) know if you are interested in collaboration. It will be the great collaboration for TWiki.

-- SopanShewale - 2009-08-14

Seems to be answered?

Closing this question after more than 30 days of inactivity. Feel free to reopen if needed. Consider engaging one of the TWiki consultants if you need timely help. We invite you to get involved with the community, it is more likely you get community support if you support the open source project!

-- PeterThoeny - 2009-10-02

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.