New Servers for TWiki.org in 2008

Sun Microsystems donated very beefy hardware to twiki.org through TWikiDotNet. Thank you very much Sun Microsystems for the very generous donation!! TWKI.NET and Plug and Play Tech Center are jointly donating the hosting cost.

Status: The new servers have been deployed on 2008-08-27.

Hardware Spec and Hosting

• Load balancer: Cisco ACE
  • Configured for round-robin with sticky sessions
• Web server 1: Sun SPARC Enterprise T5220 Server
  • Processor: 1 x 8-Core UltraSPARC T2, 1.2 GHz, 64 Threads, 4 MB L2 Cache
  • Memory: 32 GB (16 x 2 GB DIMMs)
  • Mass Storage: 292 GB (2 x 146 GB) 10000 rpm SAS Disks
  • Optical Drive: 1 DVD-RW Drive
  • Network: 4 x 10/100/1000 Ethernet
  • Power: 2 x 100-240 V AC
  • Operating System: Solaris 10
• Web server 2: Sun X4450 Server
  • Processor: 2 x Quad-Core Xeon, 2.6 GHz
  • Memory: 8 GB
  • Mass Storage: 70 GB 10000 rpm SAS Disk
  • Network: 4 x 10/100/1000 Ethernet
  • Power: 1 x 100-240 V AC
  • Operating System: 64 bit CentOS 5.2
• NAS storage server: Sun StorageTek 5320 NAS Appliance and Cluster
  • Processor: 1 Single System, 2.6 GHz
  • Memory: 4 GB (4 x 1 GB DIMMs)
  • Mass Storage: 3.2 TB (8 x 400 GB) 10000 rpm FC-AL Disks
  • Ethernet Port: 4 x 10/100/1000 Ethernet
  • Fibre Channel Host Bus Adapter: 2 Dual-Port
  • Power: 90-264 V AC
  • Operating System: Supports Solaris, Linux, Windows, HP-UX, IBM-AIX
• Switches:
  • Netgear gigabit switch (for public network)
  • Netgear 100 switch (for internal network)
• Hosting:
  • Data center of Plug and Play Tech Center in Sunnyvale, CA

Images

Sun servers, front view:

Sun servers, rear view:

Power strips and gigabit switch:

Update 2008-05-13

• we are making progress
• 2 webservers and nas backend are racked and powered up
• the sun solaris specialist who initially offered to help is mia
• so i asked someone else
• he is very knowledgeable on sys admin, solaris and load balancing
• webservers are now half way configured
• on wed night we will work on it again
• each webserver has an 8-core UltraSPARC T2 processor with 64 threads and 32gb ram
• so should be much better performance than what we have now

Update 2008-05-23

• Cisco Ace load-balancer:
  • Should be configured tomorrow Fri
• We have two networks with two switches:
  • Gigabit: Internal network (192.x) for load balancer traffic
  • 10/100: Public network (75.x) for ssh login to webservers
  • All wired up, ready for production use
• Two webservers:
  • Configured and firewalled
  • TWiki 4.2 installed
  • 6 extra plugins installed, 2 pending
  • Shared data via NFS mount from one webserver to second webserver (temporary workaround)
  • Data copied over from twiki.org (needs to be synched again on switch over)
  • Performance problem, system needs to be tuned (we asked Sun for help)
• NAS server:
  • Could not partition disk space with Java interface (possibly due to missing license?)
  • Asked Sun for help
  • We could go life before the NAS is ready (with NFS mount described above)
• Pending:
  • Configure NAS
  • Performance tune webservers
  • Install Apache module to slow down greedy spiders
  • Install 2 pending TWiki plugins
  • Patch TWiki for session tmp dir local to each server
  • Fix mail issue
  • One more data synch from twiki.org
  • Fix DNS

This setup takes much more time than anticipated. Solaris is not Linux...

Update 2008-06-03

• We had meeting last week with Sun about performance issue and NAS configuration:
  • Page load is 9 sec instead of 1-2 sec (the T5220 servers with UltraSPARC CPUs are designed for high transaction database apps, not CPU/IO intensive CGI bursts)
  • More performance tuning needed: Install Coolstack tools, optimize file system and Apache
  • Get help on NAS configuration
• TWIKI.NET is considering getting one more webserver, a fast x86 machine that works well for TWiki CGI mode
  • Currently checking with Sun on cost (possible donation?)
• Spent time today with Plug and Play specialist to configure load balancer
  • Setup basically done, but needs to be tuned and tested

Tomorrow we will spend yet one more evening to performance tune the system...

Update 2008-06-04

• We spent yet another few hours in the data center
• After fine tuning system we have now 5.5 sec page load time for small TWiki page, 16 sec for Support.WebChanges and 200 sec for Main.WebChanges. For comparison, the V240 based current twiki.org has 4 sec for small page, 7 sec for Support.WebChanges and 30 sec for Main.WebChanges.
• After installing the Coolstack tools page load time increased to 6.5 sec
• Looking into getting fast x86 machine

Update 2008-06-15

• Not so good news:
• Last week we spend again a number of hours tuning the system.
• A Sun employee specializing in web performance help profile and tune the system.
• Three key factors on speed we identified:
  • Session file check is very slow. Solved by specifying a negative value for {Sessions}{ExpireAfter}
  • Loading all users in .htpasswd is very time consuming if you have 36K users. Two suggested actions to improve performance are tracked in TWikibug:Item5704
  • Run under Speedy CGI
• With the three changes we get 3.5 sec access time with Apache ab tool, and 4.5 sec with stopwatch on browser on the TWiki.WelcomeGuest page, e.g. still way too slow for a single page. The Support.WebChanges takes 10.5 sec vs 5 sec on old V240 hardware.
• On the other hand, the servers scale really well: Almost no difference if 1 or 20 simultaneous accesses are done
• We can't roll out twiki.org on this hardware because of the page access time
• Sun has no budget now for new x86 based servers
• Peter will be away in Switzerland and Italy until 05 Jul 2008

Update 2008-07-04

• Sun has no budget this quarter to donate an x86 box to twiki.org
• TWIKI.NET will loan a quad-core x86 box with 8GB memory and 500GB RAID 1 to twiki.org (until we can swap one T5220 with a Sun x86 box)
• Machine is currently being procured
• We will install most likely CentOS, since RedHat is the most popular platform with our user base

Update 2008-08-15

• Sun is shipping the new x86 box today to the Plug and Play Tech Center data center
• ETA for switch over is end of August

Update 2008-08-21

1. NAS Storage:
   • Firmware had to be updated.
   • Login to the NAS via the Java interface is now possible
   • The hard disks are currently getting formatted (takes several hours)
2. X4450 Webserver:
   • Firmware had to be updated for OS installation via management port
   • CentOS 5.2 is installed
   • Pending tasks: Harden OS, NFS mount to NAS, tune Apache, install TWiki 4.2.2, migrate content from existing twiki.org server
3. T5220 Webserver:
   • Pending tasks: NFS mount to NAS, update TWiki to 4.2.2.
4. Cisco Ace Load Balancer:
   • System currently hangs if one webserver does not respond
   • Pending tasks:
     • Fix and test fail-over
     • Configure system to send all traffic to X4450 on low and moderate load, and to round-robin load-balance under high load
     • Load test system
   • Worst case we can go live with one webserver and NAS, e.g. without the load balancer

Update 2008-08-27

• I will switch over tonight after one more sync
• Done:
  • X4450 Webserver configured, TWiki installed, data migrated
  • NAS configured, data on shared volume
• Pending items:
  • Test and tune mod_perl. It seems to run OK.
  • Install NTP
  • Configure TagMePlugin (tags not shown)
  • Configure CommentPlugin (UserCommentsTemplate not recognized)

• Performance tests using Firebug. First number is time in sec to load page (in parenthesis time in sec to load all embedded elements on page reload (varies a lot))
  • New Sun Fire X4450 server:
    • TWiki.TimBernersLee, TWiki 4.2.2, Template login:
      • 1.4 (2.4) - local disk, 37K users
      • 1.4 (2.7) - shared NAS, 37K users
      • 1.2 (1.9) - shared NAS, 37K users, mod_perl
      • 0.5 (1.4) - local disk, 3 users
      • 0.5 (1.1) - shared NAS, 3 users
    • Main.WebChanges, TWiki 4.2.2, Template login:
      • 26 (28) - shared NAS, 37K users
      • 22 (23) - shared NAS, 37K users, mod_perl
    • TWiki.TimBernersLee, TWiki 4.1.2, Apache login:
      • 0.7 (1.3) - shared NAS, 37K users
      • 0.7 (1.3) - shared NAS, 3 users
  • Old Sun V240 server:
    • Main.WebChanges, TWiki 4.1.2, Apache login:
      • 29 (30) - local disk, 37K users, speedy
    • TWiki.TimBernersLee, TWiki 4.1.2, Apache login
      • 1.5 (2.0) - local disk, 37K users, speedy

• Musing on performance:
  • No measurable difference in access time between local storage (single disk) and NAS backend with 7 disks.
  • TWiki 4.2.2 with template login is very inefficient if there are 37K users in the .htpasswd file. Why do you need to initialize all users simply to view an open page if you are a guest or a logged in user? I think we can optimize the user management code a lot.
  • On time of Main.WebChanges: Optimization on sort by date seems to be reduced? A ls -lat Main/|head -30 takes 1 sec.
  • On tooltip: Optimization on SEARCH my limiting the scope with topic="" seems to be reduced? I had to remove the tooltip search on the Main web since it slowed down the access time of pages with tooltip to 10 sec.
  • Colas' caching solution is the next logical step. Do the cached pages update the TWiki access logs (WebStatistics)?

Update 2008-08-30

• Most of the quirks have been worked out after the server move / TWiki version upgrade.
• Issues:
  1. Session files not cleaned up. A cron job was installed as root to run tools/tick_twiki.pl, but session files kept piling up for days to over 30K files. Root does not have write access to the apache user owned files on the shared NAS. Fix: Run cron job as apache user.
  2. Memory leak of mod_perl makes server crash every 48 hours. Same issue was reported at PossibleModPerlMemoryLeak. For now as a workaround I restart Apache once every 6 hours.
     • I also observed memory leaks when I was testing HTTP engine. I think the problem are residual circular references within TWiki, so any persistent mechanism will suffer (unless it is restarted periodically, like MaxRequestsPerChild setting) -- GilmarSantosJr - 31 Aug 2008
     • Kenneth disabled mode_perl on twiki.org, a trade-ff between speed (1.2 sec vs 1.4 sec) and stability. -- PeterThoeny - 01 Sep 2008
  3. CommentPlugin template files were not recognized. Fix was to manually specify the proper system web (TWiki04x02 on twiki.org) in the {TemplatePath} configure setting. Why is the template path not fixed when you change the {SystemWeb}? That is a gotcha other people will face too. (Thanks Kenneth for the debug help!)
  4. TagMePlugin was not working properly due to template changes in the PatternSkin. Fix was to install the latest TafMePlugin. (Thanks Sopan!)
  5. Plugins get the wrong $installWeb at init time. It should be the system web TWiki04x02, but it is set to the Plugins web. This is a new bug in TWiki 4.2. Temporary workaround: Override the $installWeb in the TagMePlugin so that the proper images and css files can be found.
  6. Y% TagMePlugin tag cloud issue: The TWiki table containing tag cloud is broken due to newlines. See Plugins and PluginPackage. Fixed by adding a =separator=", " = to all tag clouds.
  7. SearchExtensions no longer working. For example, search for "import" in that topic. Seems some TWiki compatibility issue.

-- Contributors: PeterThoeny - 15 Aug 2008

Comments

It is cool to see, that there is progress in that issue.

-- MartinSeibert - 17 May 2008

So it's still hanging mid-way or...?

No offense, but am sure an expert doesn't take more than one week to setup a not-so-complicated-setup.

-- KwangErnLiew - 22 May 2008

Peter: Thank you very much for your efforts. I appreciate your contributions a lot. Thank you very much! (I am especially curious about the new TWiki and the new Plugins. They can add up to the activity in these webs here ...

-- MartinSeibert - 23 May 2008

Thank you Martin, that is the nicest thing I heard from the community in a long time. Makes up for the many hours working on open source TWiki!

-- PeterThoeny - 23 May 2008

I see for myself how difficult it is for me to contribute more than warm words which help very little. You have me deep understand und thankfulness for going deep in server configuration and helping twiki.org and that crucial points, that are truly valuable. I am looking forward a lot to a fast performance of the servers again.

-- MartinSeibert - 24 May 2008

Peter: Please update as agreed on the marketing-meeting.

-- MartinSeibert - 02 Jun 2008

looks good peter, thanks for the effort

-- CarloSchulz - 06 Jun 2008

Thank you very much for the hard work.

-- MartinSeibert - 07 Jun 2008

First off, thanks for the incredible effort and hours going into this, much appreciated

Our own performance findings in my previous job showed that SUN boxes are usually great for applications that are easily parallelized (DNS service, SMTP service, J2EE service to mention a few we offered), but that they are a less obvious answer to the demands given by "serial" applications like TWiki.

As a rule of thumb at our place response time for serial apps was divided by the Ghz factor by going x86 with this kind of app - i.e. a 1.2Ghz SUN box (as above) against a 3.6Ghz x86 resulted in a response time reduced to a third.

But what are the goals here? Personally

• I would really like to see a TWiki.org page with response times of 50ms
• I would really like to be able to google TWiki.org pages

Currently this "upgrade" is progressing in the entirely opposite direction, i.e. above we see a pending action on "Install Apache module to slow down greedy spiders" - I think this is just a terrible way to go with this hardware.

Are there any possible way we could bend this setup into a cached setup? I.e. use one of the T5220s for serving cached pages (the bottleneck would suddenly be the bandwidth, then, yay!) and the other i.e. for building content in the background? With 64 threads available I presume it could rebuild the whole of twiki.org at not unreasonable intervals (even though a page takes a few seconds to build).

I'm afraid the setup targeted right now is not going to be a good showcase for SUN - and also I am afraid that it is not going to result in a optimal user experience for twiki.org visitors either.

If I'm not totally wrong, the TWiki project does have a few caching options available already - all of which would benefit from some real life production time

Last plea: Please do consider not installing Solaris on these boxes. As you already state clearly - Solaris is not Linux!. We have lots of Linux expertise in our project, but hardly any Solaris expertise. Imho we are targeting directly our own foot by installing Solaris again - for instance:

• contributors won't easily be able to copy the production setup locally and try it out / improve it
• contributors with Linux expertise will be unable/challenged to help out on production issues
• proven TWiki-solutions for Linux will have to be re-invented for Solaris
• this list just goes on and on, really, we have lots of history here

Ubuntu is already certified for this hardware and would be an OK choice in favour of Solaris - but as most enterprises I am aware of are running RedHat as their Linux distribution of choice, CentOS would be the choice most in line with the interest of our "customer base" here.

Anyway, the fact that SUN is sponsoring the hardware shouldn't be a valid reason for making a bad community decision regarding the OS again. - If SUN is sponsoring an on-call OS support person, of course I am willing to chance my opinion

Once our project is alive and kicking again, let's re-evaluate the multi-os support argument for hosting TWiki.org at Solaris - for now we really need something that "just works".

-- SteffenPoulsen - 03 Jul 2008

Steffen, I completely agree with you analysis. See above update.

-- PeterThoeny - 04 Jul 2008

Sweet Good to see analyses on the configurations and tests!

-- KwangErnLiew - 06 Jul 2008

How does a new x86 server differ from the current solution? I don't think any amount of hardware is going to fix the performance problem. My money is on distributing twiki.org, set up local mirrors (i'm sure there's plenty of sponsors for that, i for one could do a NL/BE one, Martin already offered a DE one, etc..).

Make them read-only, with the edit links pointing to twiki.org, and a simple plugin that registers topic saves and queues them for direct replication to the mirrors.

-- KoenMartens - 07 Jul 2008

Most of the performance problems are inherent in the TWiki code itself. But adding stronger and better hardware can quickly mitigate this core issue of TWiki.

And yes, I can agree we can have a cheap-way of distributing data across, a quick rsync, and load balancing can do the job very well (as a quick fix). Of course, if security is an issue, that can be resolved at different levels too.

-- KwangErnLiew - 07 Jul 2008

I like Koen's idea.

-- MartinSeibert - 08 Jul 2008

Reading the latest updates ... do you really consider storing all txt and pub files in an NFS mounted directory accessing your NAS ??? That's no recommended setup. I would have commented along that lines on the related TWikiDotNet blog posting, alas there's no such feature there ;). Instead, consider storing txt files as near to the CPU and its RAM as possible, i.e. on the HD of the box itself and use your NAS as a backup repository only.

In addition, Colas offered help on the twiki-dev mailing list to install his caching solution. Can we pick that up, please.

-- MichaelDaum - 22 Aug 2008

As ays Michael, no need for a fast hardware, just install my cache, it was designed exactly for this use case. I can install it for you, and in which case I will also fix twiki.org to work correctly with it:

• "log in" widget will be a static link to a separate login/account page
• tags will be in view mode, with an edit tag link going into (uncached) tag edit mode
• no more included personal left bar, but a link to it

FYI, on my twiki hosted for a $20/month, it serves pages in 0.015s. Reverting back to uncached mode is one shell command.

-- ColasNahaboo - 22 Aug 2008

/me pictures another "TWiki's Lately" of a bunch of lost soles on their knees all praying to Peter "Please give us caching, please give us caching, ...", with Peter examining his fingernails.

-- MichaelDaum - 22 Aug 2008

Colas: Yes, caching is the next logical step. One thing at a time. Lack of sleep, no time to examine my fingernails. And no interest to address personal attacks.

-- PeterThoeny - 27 Aug 2008

On performance mounting all TXT files from a NAS Why did you do that? Which remote filesystem do you use? Is it NFSv3? You did measure accessing TimBernersLee which is a static topic. Have a look at the performance differences when a grep is involved, or any other TML which iterates over all topics of a web. In any case, not having the data on a local disk does have an impact, that should be avoided. The performance differences will significantly increase the more TXT files TWiki must open.

-- MichaelDaum - 28 Aug 2008

Don't know enough 'bout the technical details but it's a good thing to see twiki.org being upgraded and the new servers deployed

-- CarloSchulz - 28 Aug 2008

Uuuh, some one from the admin group should have a look at the Blog web. All links are broken due to the bug with hidden formfields. See http://develop.twiki.org/~twiki4/cgi-bin/view/Bugs/Item5922

-- CarloSchulz - 28 Aug 2008

Cool, new and faster hardware!

Some suggestions to help the new hardware to use better its potential:

• Take a look at PerlOnRedHatIsSlow. Probably, CentOS is affected too.
• rdiff does not require login anymore. Since it is a heavy operation, IMHO it should be protected from anonymous access to avoid DoS
• SpeedyCGI is a better option than ModPerl (at least to view). See comments about mod_perl at the beginning of TWikiStandAlone.

-- GilmarSantosJr - 28 Aug 2008

Thanks, Peter, for all your hard work on this.

Many people will never realise your efforts needed to make this happen, but it being complete is truly important and I commend you for it.

-- MartinCleaver - 28 Aug 2008

Hi,

it does not feel faster from my location (Germany, 1und1), did anybody else experience a performance improvement?

what about the load of the new machine compared to the old one, i.e. how is the scalability if several/lots of users that send requests simultaneously to TDO?

-- MatthiasThullner - 28 Aug 2008

Support.UserCommentsTemplate seems to have gone missing, in that the comment boxes aren't working.

-- SeanCMorgan - 28 Aug 2008

Carlo:

• TWiki.org has been patched with TWikibug:Item5922 fix (thanks Sven)

Gilmar:

• Thanks for the Perl on RedHat tip, likely same for CentOS. I will look into it.
  • I Tested the server for this yesterday and informed Peter that it indeed does suffer from the same Redhat bug - which they have known about since later last year. I suggest using a professionally supported Linux distribution. -- SvenDowideit
• rdiff requires now auth again.
• Yes, we can experiment on speedy vs. mod_perl to see what is better on twiki.org.

Matthias:

• No hard data yet, but with the old server, the load was at 4-8 during USA day time and at constant 100% CPU utilization. With the new dual quad-core server we are at load of 3-4, and at 50%-90% CPU idle.
  • I agree with Matthias, the server load is not the problem, the Bandwidth from the server's location is pittiful We Should move the servers to where the v240 is (or some other proper hosting providor). -- SvenDowideit

• Known issue. No need for a Support.UserCommentsTemplate, but for a TWiki06x01.UserCommentsTemplate that the CommentPlugin recognizes. For some reason it does not. System web is TWiki06x01. Help appreciated.

-- PeterThoeny - 29 Aug 2008

I'd rather have pages load a little slower on average than seeing Speedy back in action and the server hanging totally dead in periods. I am sure speedy was a main reason for this. Speedy speed up at low traffic but at high traffic the machine runs out of resources because of the many speedy processes.

I may have an idea on the comment problem but I have lost access to both ssh and configure so I am just a plain user with no means to do anything now.

-- KennethLavrsen - 29 Aug 2008

Had the same bad experiences with speedy. Speedy_backends simply get "stuck" with lots of speedy frontends waiting to get through to one backend on high loads. Mod_perl disappointed me too, leaving fast_cgi as the only other option. Alas, you will only get that with the help of Gilmar's TWikiStandAlone work together with the upcoming FastCgiContrib. This, however, also opens up the alley for even more optimization using mod_proxy and several TWikiStandAlone renderers on different machines. IMHO, this is the only viable option for high load TWiki sites, like twiki.org.

This means that we need a test site before upgrading and running into problems. Is there anybody willing to fund building such a test environment?

Peter, can you comment on NFS as well, please.

In any case, the new server is considerably faster than the old one (during USA night time ).

-- MichaelDaum - 29 Aug 2008

I know all the optimisation isn't done yet, but twiki.org is impressively quick this morning, and I just wanted to take the opportunity to thank Sun for their incredible generosity and proffer sincere congratulations everyone who worked on the difficult task of getting the new server up and running. It can't have been easy, with the community breathing down your necks for the last year or so.

Now, how about that new look-and-feel for the site, and a manageable taxonomy for Codev? And a mirror for twiki.org in europe, so we don't suffer another downtime like the one yesterday?

-- CrawfordCurrie - 29 Aug 2008

I am also in favor of a mirror a lot. Is that technically feasible?

-- MartinSeibert - 31 Aug 2008

Yes, full geo-distributed TWiki is feasible - I did an experiment with it several years ago, and like so many things, its a small amount of coding, and a larger amount of testing.

-- SvenDowideit - 31 Aug 2008

I need to log in again and again. The old apache-level login mechanism was nicer actually, as I hadn't to care about loggin in at all. Can we at least increase the session time out period, please?

-- MichaelDaum - 01 Sep 2008

Due to the current instability Kenneth and maybe Peter restarted the server a few times. Maybe thats why you lost your sessions.

-- OliverKrueger - 01 Sep 2008

I also experience the need to login again and again so it has not been because I restarted the server.

There are two issues. Template login behaves like this!! When I have tested with Template login I have always experienced having to login again and again. This is the reason I hate template login and always kept using apache login where my browser is able to keep the authentication.

That said I think twiki.org has an additional issue. Yesterday I noticed that the reprev is broken (the feature that prevents upreving a version when same person saves within say 1 hour). Peter found that the system time on the NAS is 6 hours wrong. I am not sure if and when this has been corrected.

This could also affect session files. The session expiry time is set for 6 hours on t.o as I recall and it handled by cron job.

If the session files have wrong time stamp for sure we will see a huge issue.

-- KennethLavrsen - 01 Sep 2008

This was the third trip to the data center on this extended weekend.The time on the NAS was set to local time instead of GMT. This is fixed now. I verified that the version bump-up issue is now fixed. The session problem of re-login should be fixed as well.

Please report any other issues you might find here.

-- PeterThoeny - 01 Sep 2008

Peter, can you please explain why you use a network filesystem on a NAS to store txt files, and as I have learned now, temporary files as well?

-- MichaelDaum - 02 Sep 2008

I just noticed an update of WebStatistics made by TWikiGuest. I tested it (force an update without been logged) and it worked. Maybe it should be restricted to authenticated users only, like rdiff.

-- GilmarSantosJr - 17 Sep 2008