create new tag
, view all tags
InterfaceThread - FeaturesThread

We have noticed that often our proxy/cache serves up out of date wiki pages. Not a great situation where edits are coming thick and fast.

Let's add a couple of lines to the HTML header...

<META HTTP-EQUIV="Expires" CONTENT="Mon, 06 Jan 1990 00:00:01 GMT"> 
<META HTTP-EQUIV="Pragma" CONTENT="No-cache">

No, it doesn't have to be the 6th Jan 1990 smile - but a date/time in the past gets the proxy to discard the page even if it overrides the No-cache Pragma.

While on the subject, how about some advice to robots...

<META NAME="Googlebot" CONTENT="Noarchive">   
<META NAME="Robots" CONTENT="Index, Follow">
<META NAME="Revisit-after" CONTENT="14 days">

And if we are feeling in the mood to extend the functionality...

<META NAME="Author" CONTENT="[...a list of contributors to the topic]">
<META NAME="Keywords" CONTENT="[...perhaps we could add the ability for
  editors to add/change Keywords?...store as METADATA???]">
<META NAME="description" CONTENT="[ditto]">

-- SteveRoe - 19 Jun 2001

Ahhh...I have now spotted the following variables:


So - in theory, I can just set these in WebPreferences for the Codev web. Let's try!

Wow...that was a doddle! Does anyone mind me doing this? Have any other opinions about the values to use???

-- SteveRoe - 19 Jun 2001

1) Can it be used in conjunction with the Release Edit Lock (UnlockTopic) check box to disallow someone to back-arrow in the event that they said they didnīt want to edit the topic again?

2) Is it going to prevent people taking an offline copy using Favourites | Add to Favourites ! Make available offline ?

-- MartinCleaver - 19 Jun 2001

Steve, thanks for the initiative, but I decided to undo the META expire settings in WebPreferences for these reasons:

  • Not everybody is behind proxi/cache servers.
  • Doing a frequent action like going back to the Changes topic is very slow because the page gets reloaded every time. This is problematic when accessing TWiki.org over a dial-up line.
  • All changes get lost when going back in preview to make more changes!
  • For people who prefer the META expire settings: Add them to your home topic as personal preferences. You need to log on (e.g. edit a page once) in order to make this happen (TWiki will remember you based on the IP address)

-- PeterThoeny - 19 Jun 2001

Ah! Sorry for jumping the gun. Will try the personal settings. I am a little concerned that the IP address mechanism will get confused by any NAT that we may have. Let's try...

...OK - this works for me - and does not seem to be too bothered by NAT even when we try with two different client machines.

-- SteveRoe - 20 Jun 2001

On your points:

  • I understand that you do not want to regen a changes page when hopping back and forth - but equally, I suspect that many users do not want to accidentally view an out of date changes page. In other words there may be a reasonable expiry time (something similar to the lock expiry time???) that gentlemen can agree on. I suggest that when we get around to it, have datetime addition in the TWiki variable set and can agree on the number we add this to the TWiki org site.
  • Obviously whoever added the %HTTP_EQUIV_*% variables was smart enough to recognise the need for different behaviour on view, preview, edit! My excuse is that IE does not reload an expired page on Back, whereas NS (4) does. Obviously a major benefit of IE (only kidding!) :).

-- SteveRoe - 20 Jun 2001

Steve & others - have a look at BrowserIssues, where I've documented some browser cache issues. You may find the Netscape 4.x behaviour can be configured to match IE. Also, it would be good to document these sorts of ProxyCacheIssues as a sort of hints & tips item to go in the TWikiDocumentation topic.

-- RichardDonkin - 20 Jun 2001

I was surprised to see that the following line...

<meta name="robots" content="noindex" />
...now made it into view.tmpl on TWiki.org . I suggest that we encourage robots to index pages that are accessed by view in order to make the wiki contents accessible to search engines. Certainly we should discourage indexing of preview, attach scripts, etc. I guess that I had thought that this was the purpose for having different variables for HTTP_EQUIV_ON_VIEW, HTTP_EQUIV_ON_PREVIEW, etc.

-- SteveRoe - 19 Jul 2001

This is not the case for view of the top revision (unless there is a bug). Only view of older topic revisons have the "noindex" meta tag, among other templates.

-- PeterThoeny - 19 Jul 2001

Aha! All is now clear - thanks for the explanation!

-- SteveRoe - 20 Jul 2001

Speaking as someone who deals with proxy caches day in day out, and troubleshoots and fixes these particular sorts of problems on a regular basis (mainly to help scalability), I'd say that we: a) Want to produce web pages friendly to proxy caches. b) Want to make sure that the output people expect is what they expect to see.

For example many dynamic websites run with reverse proxies to reduce the load on the back end servers by a huge amount. For a proxy cache, the key factor that most people misunderstand is that a proxy cache does not look at the HTML content . It only looks at the *HTTP* headers.

There are two main approaches to increasing site cacheability:

  • A Make an effort to mark the content explicitly as cacheable and put controls on how long something is cacheable.
    • This really means generating things like Expires, or cache control headers.
  • B Go out of your way to not produce content that looks uncacheable. This means:
    1. Include/generate a Last-Modified header
    2. Respond to If-Modified-Since requests properly and send a 304 not modified if the document hasn't been modified since that date.
    3. Avoid the use of the letter sequences "cgi-bin", "asp" suffixes, and punctuation that dynamic content normally has - eg question marks, semicolons, and so on.

Locally I've patched our twiki server view code to do B1, since Twiki generally does B3 already. What I haven't done yet is add in support for 2 yet. If you do this page caching is dead simple - just put a reverse proxy in place - since they're good at that sort of thing... The lack of 1 & 2 effectively results in all data from a Twiki web being non-cacheable - which is bad for a Twiki server...

Why is the above sufficient? In the B case, caches generally work on the basis of documents that change often have a low age (calculable from the Last Mod data), and documents that change less often are older. eg Suppose you have 1200 pages that change once every 10 minutes, but a page changes every second. This means if you take the average age of all the documents is around 5 minutes. Likewise similar stats will hold for pages that change every 10 seconds or every 10 days.

As a result most proxy caches do this:

  • Find out the age of the document when downloaded
  • Multiply this by a small factor - eg 1.1, and state that when the document reaches that age, that the proxy cache must revalidate the document.
  • As a result if a document is 1 day old it will be valid using the above rules for just under 2 1/2 hours.
  • If the document is 1 hour old, it will be viewed as fresh for 6 minutes.

As I say, meta tags have no effect what so ever on a proxy cache's controls of cacheability. Meta tags do control browser behaviour, which is just as important to deal with.

FInally regarding Peter's comment above:

  • Not everybody is behind proxi/cache servers.

True. However the figure is likely to be much higher than you'd expect.The majority of ISPs and many carrier networks, and relatvely large numbers of business are behind on proxy or another whether they know it or not, since many view it as integral to the network these days. I'd estimate that at upto 50% - 60% of people go through one form (or brand smile ) of proxy cache or another.

Since some browsers (eg IE) behave very badly with transparent caches that don't know that IE deals badly with transparent caches, having controls on caching in Twiki that control both a proxy cache and the browser is a GOOD thing IMO, especially if controllable from Web Preferences. Once I've finished adding the extensions necessary, I'll roll the patch back to the community...

-- TWikiGuest - 20 Jul 2001

See BrowserAndProxyCacheControl for some more discussion on this area, and links to tutorials and cache testing tools. I've just managed to fix a long running browser-related caching issue (described in BackFromPreviewLosesText), so I'm quite interested in this whole area!

-- RichardDonkin - 20 Jan 2002

Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r13 - 2003-09-06 - MichaelSparks
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.