Would Precompiling be a good idea ?

If you look at the various WebStatistics pages, you'll noticed that Views make up at least 90% of the 'action'. (Count save as at least two actions save plus edit, though this ignores the preview cycle which general involves much rendering.)

So from a purely user usage profiling side of things there seems some value in decreasing the view time.

Some precompiling or preparsing of the topic files might be worth considering.

I'd be interested in what sort of view:save/edit ratio other TWiki installations get.

-- NicholasLee - 05 Aug 2000

We get 70%-90% of traffic as views

On the my dev site it's about 70% view, 30% everything else. On my production site it's 92-97% view (although a really hot topic can drop views down to 85% or so; apparently there are people who only look at a topic once when they edit. This site has only six users, which certainly colors those numbers.)

-- KevinKinnell - 07 Aug 2000

Read Only Sites? Main Web was 5500:160 read:writes

I guess that is one thing we'll have to consider for ReadOnlySites. Consider WebStatistics. 5500 reads to 160 writes in July. WebHome is probably quite indicative of an active TWiki.

-- NicholasLee - 07 Aug 2000

Sounds nice, how about precompile on write?

Keeping a precompiled (I'd prefer to say pre-rendered) version of the page increases disk space, but I agree that the savings in time would be quite worthwile.

Precompling could be done when the page is modified. But there's a case when a page needs to be precomipled without having been changed: for instance, if we have a nonexisting topic reference (ex: BlahBlah), we'd have to know that when the BlahBlah topic was created, this page would have to be precompiled. That could be easily fixed if TWiki had BackLinking.

-- MarcoCarnut - 06 May 2001

Dynamic Pages are TWiki's advantage, keep non-compiled as well

ok, I started with a JOS wiki (which did this kind of thing), and we went to Twiki because we liked the idea that pages are dynamically generated, allowing us to have queries on a wiki page automatically updated by re-freshing. This can be catered for by having a No-Cache marker on a page, but ....

my second thought on this, is - do you have performance problems? Just because something happens much more often than another, doesn't mean that it is a problem

I may have been jaded by the fact that Unisys gave me a dual Xeon box for the wikiserver while i was there and my PIII 850 is way fast enough for my personal wiki.

-- SvenDowideit - 06 May 2001

Sounds Useful, but problems with dynamic things?

I do have a slow box with enough space, so precompilation would help me tremendously.

OTOH it's quite difficult to do this well. TWiki would have to reliably track all pages that depend on another page. This goes beyond links; things that come to my mind are:

• Link rendering
• Included files (such as SiteMap)
• Environment variables (like background color)
  • Environment variables in pages
  • Environment variables in wikicfg.pm .
• Some pages depend on every topic in the TWiki, such as
  • Index (should still be cached)
  • Search results pages (these should be non-cached)
• User settings
• User specific rendered topics

I'm pretty sure that I overlooked some things; somebody with a better grasp of the rendering engine should look up what changes may have an effect on rendering.

-- JoachimDurchholz - 07 May 2001

This is how we do it. No problems, but dynamic version always available

We do this using the GenHTML plugin. From a cronjob, the following script runs every 15 minutes:
(Hostname changed to 192.168.0.1 and webname changed to Stuff to protect the innocent)

#!/bin/sh

lynx >/dev/null 2>&1 -dump 'http://192.168.0.1/twiki/bin/genhtml/Stuff/HomePage?webname=Stuff&exclusions=Web.*&action=Generate%20HTML'

The primary URL that people access twiki via is http://192.168.0.1/ - this is achieved by pointing the docroot for that virtual server at <twikiroot on filesystem> /webs/Stuff/pub/HTML/, on a twiki that doesn't implement data and code separation, the docroot would obviously be: <twikiroot on filesystem>/pub/Stuff/HTML/ .

Gotchas:

• The Web.* exclusion excludes WebHome as well. Whilst we could list explicitly pages to exclude, changing WebHome to HomePage is simpler - this requires configuring in TWiki.cfg
• A symbolic link in the HTML directory from index.html pointing at HomePage.html is needed to ensure that things work as you'd expect.
• You clearly need to add an edit (and "more") button onto the published/precompiled versions. Edit points you at the edit URL for that page, the "more" button takes you to the non-precompiled version.

The nice thing about this approach is that it's so easily extensible I also use this approach for 2 personal domains as well on a personal server. (One of which is http://www.lansdowneresidents.org/ )

(In the above, hostname changed to 192.168.0.1 and webname changed to Stuff to protect the innocent)

Clearly this forces everyone to use the same skin for viewing (which isn't the default), but in our environment this isn't a problem - the speed benefits outweight the downsides. As far as I can tell none of the issues Joachim raises above have been an issue - even things like the comment plugin continue to work and Do The Right Thing.

-- TWikiGuest - 21 Jun 2003

Michael,

Wonderful! Thanks!

-- RandyKramer - 22 Jun 2003

before Michael's proposal gets implememted i need to say - this will not work for those of us using the twiki for dynamic pages based on TwikiUserName and user settings. In my Bugs system the MyBugs topic has a single query on it that will show that user's bugs. and i think there are a number of other pages like this. at minimum you need to provide a way of marking topics as non-PreCompilable (and in such a way that users understand that they need to set it at the right time.)

the propsal also removes the user settings for twiki rendering (like SKIN = tiger), and

.... errrr....

can i simply suggest that its a bad idea to load your mahine heavily every 15 minutes so that the web server doesn't respond to requests for a bit?

why not work on optimising the rendering loop instead?

-- SvenDowideit - 23 Jun 2003

Issues:

• User settings for skins overridden - On our site (and many other installs I've put in place) I've yet to see more than 1 or two per hundred users actually bother changing their skin. (This is an over estimation of the number of users I've seen using different skins) If they do however and want to access the site that way, they don't have to look at the pre-compiled version. Whilst this isn't the same as our work wiki, it's done using "similar tricks"... compare & contrast:
  • http://www.lansdowneresidents.org/ (similar to our precompiled version - if it's slow, it's because it's on the end of a DSL line!)
  • http://www.lansdowneresidents.org/twiki/bin/view/Lansdowneresidents (non-compiled version)

• Personalised "MyPages" - We don't have any of these at present so it's not a problem for us. The generic solution would be this:
  • Deem certain pages not-compilable, but linkable
  • Deem certain pages not-compilable, and not linkable
  • This then allows "MyPages" to be linked to by users, and when people access them, they're taken to the dynamic version. (Perhaps even a meta-equiv-refresh on a compiled version to take you instantly to the dynamic version)

• Bad idea to load machine every 15 minutes, optimise? Currently this isn't an issue on the machines we're running this on - even when a rebuild is in progress (after all each page generated is like a sequential view without the TCP connection creation overheads). As a result I've not bothered changing the GenHTML script (which isn't maintained and says so in BIG BOLD WORDS ). I might look at the publishing addon at some point in the future to see if it does things simpler. For reference though we also produce PDFs of every page on the site from a script that runs every 15 minutes. That however only generates PDFs for documents that've changed since last run.

  Bottom line is I've only optimised things that need optimising at this time. At some point it will need looking at again, but for a 1/2 hour hack about it gave a BIG speedup.

I don't think this stuff should be included in the core per se, I was simply suggesting how people can do these things right now, without core-code changes, and without major problems with a little care.

-- TWikiGuest - 24 Jun 2003

A solution that would require some modifications to the core would be to have all reading of files done while rendering a page be logged. Thus saving or modifying these files would invalidate the cached page. The loaded pages would be the prefs files, included topics, web directories themselves (their dates) for %SEARCHes, etc... You would log also tentative access to non-yet existing files so as to make the ? disappear when a topic is created after being referred to.

It is not rocket science, but implies going through all the code to check what we are doing, and provide a "cache management" API for plugins to use to register also their file access. An function to invalidate cache should also be provided to be called after modifying files not from inside TWiki (template files, TWiki.cfg).

The tricky part is that the "sources of information" are not just files but also variables (%USERNAME, %TIMEOFDAY...) and the system should be modified to log the use of these variables, which may not be easy to do.

This solution I clean, I think, but involves lots of small changes and some clear-minded desin and code... not trivial.

-- ColasNahaboo - 24 Jun 2003

Category: TipsAndTricks