Replication Using Rsync

One of a set of ReplicationTechnologiesForTWiki

Rsync

Rsync was used to great effect at Inktomi many years back for a significant time period (>12 months of active use) to replicate data between sites, and worked very well. One limitation of the approach taken was that you could only have one master site for each web , but that was resolved by simply making the edit and attach links point at the appropriate master site for that web. Rsync is designed to be lightweight and replication between 5 systems on 2 continents worked very well and transparently. (To the extent that the preferred mechanism of shipping content between sites for certain tasks became attaching the content to a TWiki page, and letting the "TWiki Net" as people referred to it perform the replication. Furthermore this was despite the fact that Inktomi also sold content distribution & synchronisation software.)

-- MichaelSparks - 14 Jul 2005

• perl cpan and rsync - http://search.cpan.org/~cbarratt/File-RsyncP-0.52/lib/File/RsyncP.pm might help.
• Using rsync over ssh - http://hacks.oreilly.com/pub/h/38 , http://www.mikerubel.org/computers/rsync_snapshots/
• Rsync windows bundle - http://www.itefix.no/phpws/index.php?module=pagemaster&PAGE_user_op=view_page&PAGE_id=6&MMN_position=23:23
• Rsync from CygWin Windows to Linux: http://www.gaztronics.net/rsync.php

-- MartinCleaver - 19 Jul 2005

Our TWiki installation uses a slightly different approach. We are using the September 2004 release, which had some left-over mirror code within it. We made a few modifications to this code as needed, only amounted to a few lines in three or four files, and changed the pattern skin templates. TWiki.cfg on each server now defines its location, and one web is set to be "master". Each mirror web has all editing commands (Edit, Attach, etc.) struck-out and unavailable. We added a "Select your mirror" pull-down to the top bar so that people could view their local web, or access the master for edits. A 15 minute rsync scheme where the slaves pull from the master keeps everything in sync. We're managing 10G+ of data, three mirrors and a master site using this scheme. Contact me if you would like any more detail.

-- IanHarrison - 03 Jan 2006

yes please Ian while we may not end up using your changes, it is always interesting to see what changes people make, and how code (and stubs) are used. I'm not sure if the mirror code has been tested in DakarRelease, as very few people know about it, and fewer use it...

could you post a diff, and any other interesting tid-bits?

-- SvenDowideit - 03 Jan 2006