Question

Since a couple of days (since persistent perl installed?) the pages on twiki.org do not load completely. Seen on Safari and Firefox Mac.

-- ArthurClemens

Environment

-- PeterThoeny - 07 Sep 2006

Answer

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.

It might also be a problem with the deflate mechanism? I think for attachments there was kind of the same problem when they were being deflated; some times you would get all of the attachment, sometimes only a chunk of it.

-- SteffenPoulsen

I created this entry based on Bugs:Item2846

-- PeterThoeny - 07 Sep 2006

I experience this quite often on FF 1.5.0.6, possibly 5% of the time, as seen in this screenshot of partial load. Reloading the page once or twice fixes the issue. It started at the time when we enabled SpeedyCGI on twiki.org.

-- PeterThoeny - 07 Sep 2006

I see this quite often on Internet Explorer as well. It does not seem to be browser dependent. It is actually quite annoying.

It seems this SpeedyCGI thing is not so wonderful after all.

-- KennethLavrsen - 08 Sep 2006

We have used SpeedyCGI consistently for view since 4.0 and we never experienced this (Linux).

On another note, cefore SpeedyCGI was installed at twiki.org there was sometimes (perhaps 1 for each 50) a similar problem when editing a page, the problem being that the page was not loaded fully. A refresh also cured this (there are some mentions of this problem in IRC as well, i.e. http://koala.ilog.fr/twikiirc/bin/irclogger_log/twiki?date=2006-07-29,Sat&sel=253#l249).

The reason I'm mentioning this is only that I'm not really convinced that the problem is due to speedy, we should at least test what effect disabling deflate gives us (my imagination so far only suggests this as alternative cure).

Current speed of twiki.org is nice - but problems with presentation are of course not acceptable.

-- SteffenPoulsen - 08 Sep 2006

yeah, sadly, i'm of the opinion that we're just seeing a pre-existing issue, thats made more common with speedy installed. I'll continue to look..

-- SvenDowideit - 08 Sep 2006

I have experienced this at least 20 times just today. It is really annoying. I had to reload one topic 3 times just to read it.

Do we want twiki.org to continue like this? Or do we disable speedy again until we know what is wrong? Twiki.org is our face to the world.

-- KennethLavrsen - 14 Sep 2006

OK, agreed, I disabled speedy cgi for now...

-- PeterThoeny - 14 Sep 2006

I re-enabled it with different parameters:

• old: -- -M10 -r128 -t3600
• new: -- -M10 -r50 -t600

Thanks to MichaelDaum's recommendation. Let give it a try and see if this works better.

-- PeterThoeny - 15 Sep 2006

I still have incompletely loaded pages ocasionally.

-- ArthurClemens - 16 Sep 2006

How about disabling mod_deflate and see if this makes any difference. There could be some ajax related problems too. So just to nail the issue down we should also disable google ajax search for a while just to bring TWiki.org down to a more common setup.

-- MichaelDaum - 16 Sep 2006

For testing I disabled mod_deflate. Lets see how this affects page display and server load.

-- PeterThoeny - 19 Sep 2006

my experience is that turning mod_deflate off decreases the total number of twiki topic requests the server serves by about 30% - very little change in cpu, and obviously an increase in bandwidth used.

and you can see what looks (its way too early to tell though) like that at http://twiki.org/~sdowideit/mrtg/twiki/twiki.html

-- SvenDowideit - 19 Sep 2006

Definetely helps here, my pages are complete now (the exception being Codev.WebChangesForAllWebs, which still only loads ~½ for me for some reason). Is anybody aware of known incompatibilities between speedy and mod_deflate (I haven't been able to google any)?

-- SteffenPoulsen - 19 Sep 2006

It helped a bit, but I still get incomplete pages once in a while. So the problem might not be related to speedy + mod_deflate.

-- PeterThoeny - 20 Sep 2006

I'm still getting many half loaded pages on twiki.org and Bugs Web on develop.twiki.org has disappeared too. It's not my day.

-- FranzJosefSilli - 21 Sep 2006

I also get messed up pages, and if you look at the mrtg twiki requests url above, you can see that there seems to be a net increase in topics served. So turning off mod_defalte has increased our throughput (yay), and increased the bandwidth we use (almost double )

wrt the Bugs web, I presume that you are still using the ~develop version rather than the ~twiki4 version (the ~develop one was removed on a decision in the last meeting *LINK* )

I've just added Redirect /~develop http://develop.twiki.org/~twiki4 so the original Bugs url will continue to work at least until we kill DEVELOP more permanently.

-- SvenDowideit - 22 Sep 2006

I disabled disk caching of /cgi-bin/view (provided by experimental mod_cache Apache module). Let's see if this fixes the partial page load issue; let's see if this has a performance/load impact.

-- PeterThoeny - 22 Sep 2006

For me there is no change since my last report. Small pages started to display OK after the mod_deflate change, large pages are still mostly broken.

Using Codev.WebChangesForAllWebs for testing: When it breaks I seem to typically get just above 70 kilobytes (example results: 78848, 77824 or 75776 bytes) before the connection is forcefully closed. Do you get similar byte ranges? (Page at the time of testing was 94818 bytes; 78848 bytes is ~83% of this)

Could be a race condition related to multi-CPU setup? Could be a race condition related to inter-communication thread space? Can Apache perhaps be compiled in a more tight way to handle this kind of stuff in an alternative/more restrictive way?

Something else: Am I right in presuming that adding SpeedyCGI was the only change made to configuration at the time this error started to occur, or were there perhaps other changes to the setup at the same time?

-- SteffenPoulsen - 22 Sep 2006

no, you're right, the only thing i did to enable speedy was to change the #! line on the view script - speedy had been installed quite a long time before that.

-- SvenDowideit - 22 Sep 2006

Yes Kenneth, if I enable SpeedyCGI the issue show up; it is OK when I disable it. Must be some combination of OS, software versions and configurations.

Sven, could you please install the latest PersistentPerl version?

-- PeterThoeny - 25 Sep 2006

from my investigation we are running the latest.

from Solaris pkg-get: pm_speedycgi         2.22,REV=2006.02.11                        SAME

from http://daemoninc.com/SpeedyCGI/download.html :

 SpeedyCGI Download
    * Version 2.22, Sat Oct 11 20:34:15 PDT 2003

http://daemoninc.com/PersistentPerl/download.html

 PersistentPerl Download
    * Version 2.22, Sat Oct 11 20:34:15 PDT 2003

The error did not start to occur when we turned on SpeedyCGI, it was present before that, just much much rarer. I remember getting partial loads during the month before too, when I was getting large numbers of topics, there would often be one 'dodgy one.

-- SvenDowideit - 25 Sep 2006

Oh, yes, we are running the latest version 2.22.

Actually, I have not seen the partial page load issue with disabled speedy. Replace /view/ with /viewauth/ on any topic that loads partially. The viewauth script uses regular perl, and it works every time.

Some more data:

• ClassicSkin works OK, partial page error with PatternSkin (append ?skin=classic to view URL)
• Google Ajax search does not seem to make a difference
• Type of content does not seem to make a difference: A large page Sandbox.WebChangesForAllWebs with SEARCH (e.g. external grep program call) shows the error, also when cached with VarCachePlugin. Cached content is like static content (when no refresh is done)

-- PeterThoeny - 26 Sep 2006

This weekend I have had to reload pages constantly because of incomplete loading. I had to give up twice because even after 5-6 reloadings the page did not arrive. Even a short simple topic was incomplete.

-- KennethLavrsen - 01 Oct 2006

I suspect that this is caused by Apache setup (compilation/configuration) combined with speedy.

-- PeterThoeny - 01 Oct 2006

I'm having a lot of trouble with that too, see my post on TWikiDotOrg.

-- StephaneLenclud - 03 Oct 2006

Wild guess: Is speedy compiled with or without the WITH-THREADS option?

-- SteffenPoulsen - 03 Oct 2006

This is happening very frequently on TWiki.org at present, and sometimes even reloads don't work. Happens on both the sidebar and main part of page.

How about trying ModPerl, which has been used quite a bit with TWiki and may help reliability? If this could be done under a separate /modperl/ URL while enabling SpeedyCGI still for /cgi-bin/ that would enable parallel testing. Or just switch over, it's not too hard to switch back to speedy if needed.

-- RichardDonkin - 11 Oct 2006

It appears to be fixed now, and TWiki.org seems much faster these days!?!

-- StephaneLenclud - 24 Nov 2006

I disabled SpeedyCGI on 2006-11-05 due to intermittent server errors. Pages load now normally most of the time, although I can reproduce partial page loads if I on a slow dial-up connection.

-- PeterThoeny - 26 Nov 2006

I'm about to create a mod_perl test location, but in the process noticed that the speedycgi package was upgraded - so I've turned it on for a weekend test.

-- SvenDowideit - 28 Apr 2007

I'm getting a lot of 500 before I can eventually upload any plug-ins these days.

-- StephaneLenclud - 28 Feb 2008