RCS seems to have a limitation on the file size. We have now around 7000 registered users at TWiki.org. The TWikiUsers topic gets updated automatically with each registration.

-rw-r--r--    1 nfsnobod nfsnobod   228780 Sep  4 20:42 Main/TWikiUsers.txt
-r--r--r--    1 nfsnobod nfsnobod  1972295 Sep  2 04:11 Main/TWikiUsers.txt,v

Registration does update the TWikiUsers topic, but since a few days ago, the rcs file does not get updated anymore. Possibly there is a 2MB limit on the rcs file, or a 220KB limit on the source file. The man pages do not mention a limit. Anyone knows? Anyone knows a fix?

-- PeterThoeny - 05 Sep 2003

Strange - it could be an RCS bug (perhaps with large number of revisions?) but a quick search didn't find one.

I did check ulimit -a on twiki.sf.net, which hopefully has similar setup to the web servers, but a CGI script would tell for sure - the -f value limits file sizes but is much bigger than this, so it's not a Linux limit:

rdonkin@sc8-pr-shell1:~$ ulimit -a
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) 32768
file size             (blocks, -f) 262144
max locked memory     (kbytes, -l) 32768
max memory size       (kbytes, -m) 32768
open files                    (-n) 50
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 8191
virtual memory        (kbytes, -v) 262144

-- RichardDonkin - 05 Sep 2003

We have now one more limitation at TWiki.org: The ArgumentListIsTooLongForSearching the Main web with over 7500 topics.

-- PeterThoeny - 10 Sep 2003

Two minute test hack:

#!/bin/perl

open(IN, "<iter.txt");
$x = <IN>;
chop $x;
close(IN);
$x = 1 if !$x;
while (1) {
    print "$x\n";
    open(OUT, ">iter.txt");
    print OUT "$x\n";
    close OUT;
    system("curl",
        "-u",
        "jcline:pass",
        "-F",
        "text=<iter.txt",
        "http://localhost/~jcline/cgi-bin/save/Main/TestTopic"
    );

    $x++;
}

However, I had to manually set $doUnlock in Store.pm at top of saveNew in addition to setting the cfg vars $editLockTime = "0"; and $doKeepRevIfEditLock = "0"; and I'm not sure why (I don't think it's the + 120 secs that are added either).

Test Results (on FreeBSD 4.7 server using rcs v5.7):

• Up to v1.9168 and still running ok:

-rw-r--r--  1 www  www      125 Sep 10 22:01 TestTopic.txt
-r--r--r--  1 www  www  2001743 Sep 10 22:01 TestTopic.txt,v

• Up to v1.34390 and still running ok:

-rw-r--r--  1 www  www      127 Sep 11 08:49 TestTopic.txt
-r--r--r--  1 www  www  7649317 Sep 11 08:49 TestTopic.txt,v

• Up to v1.43299 and still running ok (well.. except for an apache problem, see below .

-rw-r--r--  1 www  www      127 Sep 11 15:28 TestTopic.txt
-r--r--r--  1 www  www  9643593 Sep 11 15:28 TestTopic.txt,v

• Up to POST attempt 64351 and revisioning has temporarily stopped at v1.64168

-rw-r--r--  1 www  www       127 Sep 12 12:50 TestTopic.txt
-r--r--r--  1 www  www  14315113 Sep 12 12:50 TestTopic.txt,v

• • I believe revisioning stopped due to another restart of httpd while the script was running. The lock on the rcs is owned by admin. Perhaps there's a race condition when starting (or stopping?) apache and POSTing, leaving apache with a wierd uid. Editting in TWiki and restarting the script has everything running properly again.
• Up to v1.66077 and still running ok--the 16-bit boundary (65535) passed ok! (Note: rdiff ran for over 25 minutes before I killed it and was only able to render a couple dozen diff's!)

-rw-r--r--  1 www  www       127 Sep 12 16:13 TestTopic.txt
-r--r--r--  1 www  www  14745870 Sep 12 16:13 TestTopic.txt,v

• • This is getting a bit obnoxious at this point but what the hey, I'll let it run over the weekend. :-P
• Up to v1.115872 and still running ok--the six-digit boundary (v1.100000) passed ok.

-rw-r--r--  1 www  www       129 Sep 29 11:15 TestTopic.txt
-r--r--r--  1 www  www  25979237 Sep 29 11:15 TestTopic.txt,v

• I'd guess it's a sourceforge limitation.
• The above ,v file size translates into 218 bytes per revision. The file content itself is only 5-7 characters (a text number). That's over 3000% overhead. What a waste. Look what happens when I try gzip -1 on the ,v: 197913 bytes; that's a compression ratio of 90.5% ! So it'd be quite an improvement to compress after store; use a pipe to eliminate the temporaries (that'd likely cause oddities in cygwin).
• Speed at v1.38294 is about 5x as slow as v1.1. (Subjective measurement + looking at seconds between revisions; it's about 3-4 seconds right now, whereas previously it was 1 second-- the minimum resolution of RCS.)
• At around v1.43270, the IT guy came in my office and asked why the server's /var was at 98% capacity (over 250megs), though du didn't show any real usage on that partition (about 2.5 megs). I guessed that the apache log had grown a bit too big, had failed to rotate/truncate, and perhaps didn't exist as a valid filehandle or something, and sync didn't flush it. Turns out /var/log/httpd.log didn't exist (it should); restarting apache solved that problem and the non-existent 250 meg filehandle was flushed (/var back to 1% capacity). So I guess TWiki is more stable than apache in this case. (My log.txt has grown to 3.7meg.)
• At around v1.82313, my office machine Windows XP hung, so my ssh session to the server was terminated. Thus I conclude that TWiki is more stable than Windows XP. (Though perhaps that's a rather sad comparison, with Windows being an incredibly unreliable piece of software.)

-- JonathanCline - 11 Sep 2003

Thanks for all the testing - this at least shows that it's specific to SourceForge, and is a great stress test!

Could you post details of your OS and RCS version, and whether the OS included RCS or you got it from elsewhere? It may be worth compiling up the latest version of RCS on the SourceForge servers and using that.

• See above. -- JonathanCline - 15 Sep 2003

-- RichardDonkin - 13 Sep 2003

Thanks - you didn't specify whether RCS is built-in to FreeBSD but it turns out that it is. The CVS source browser is here (assuming RELENG_4_7_0_RELEASE is the right tag), and it seems that it's a port of GNU RCS, so it's quite likely this would work on the TWiki shell servers, since the original code is quite mature anyway.

-- RichardDonkin - 16 Sep 2003

I am setting this to resolved, based on the extensive tests.

-- PeterThoeny - 23 Jul 2004