%META:TOPICINFO{author="BaseUserMapping_333" date="1303184385" format="1.1" reprev="1.4" version="1.4"}%
---+!! GitPlugin
%TOC%
---++ TWiki on Git Overview
The goal of this plugin is to enable a distributed operation of
TWiki. Multiple TWiki installations may share some or all of their
webs such that these webs are synchronized with each other.
Synchronization of webs is achieved by leveraging git as the
repository for TWiki topics.
Git is a distributed version control system. This plugin uses Git as
the Store module, replacing the standard RCS module. The store module
controls the storage and revision control of topics and attachments.
Git will then also distribute any topic and attachment under its
control to all other synchronized TWiki installations.
Connection between each site is through =ssh=, a secure mode of
accessing remote sites.
In the current version, two modes of operations are supported:
Distributed Mode and Centralized Mode.
---++ Distributed Mode
All sites are symmetrical in this mode. Each site has to fetch the
content of all other and merge with its own content.
---++ Centralized Mode
A central repository is used to store a copy of each site.
Each site then fetches the content of the other sites from the
central repository and merges with its own content. Each site
pushes its content to the central repository.
The centralized mode is advantageous if the sites cannot connect
directly to each other, e.g., if they maintain firewalls that do not
allow ssh connections. In this mode, the individual sites only need to
be able to connect to the central repository.
---++ Installation
The installation of the %TOPIC% primarily involves setup of the git
repositories and the communication environment, which is described in
this section.
The following terminology is used throughout these instructions.
* =<localserver>= refers to the local TWiki server. A user account =<localuser>= is assumed to exist.
* =<remoteserver>= refers to any remote TWiki server. A user account =<remoteuser>= is assumed to exist.
* =<centralserver>= refers to the central repository server. A user account =<centraluser>= is assumed to exist.
* =twiki= refers to the parent directory of the =data= and =pub= directories of each TWiki installation.
* =<localname>= refers to an arbitrary name given to the local repository.
* =<remotename>= refers to an arbitrary name given to a remote repository.
* =<centralname>= refers to an arbitrary name given to the central repository.
* =<centralrepo>= refers to the path to the central repository.
* =<remoterepo>= refers to the path to the remote repository.
* =<localrepo>= refers to the path to the local repository.
Note that the local and remote repositories must be the the parent directory of the =data= and =pub= directories of each TWiki installation. The central repository need not be a TWiki installation.
To begin, TWiki should be installed as normal. Make sure it works well
before attempting to install %TOPIC%. It is assumed that at each site,
the user indicated above, such as =<localuser>= will execute
TWiki.
These instructions assume a linux distribution for the TWiki server.
---+++ SSH Installation
=ssh= public and private key should be prepared for each server. For
example, you can log in to =<remoteserver>= as
=<remoteuser>= from =<localuser>@<localserver>=
without a password. Keep the private key in a directory where
=<localuser>= has the right to access it, but ensure that the
permissions satisfy the requirements of =ssh=.
<localuser>@<localserver>$ssh-keygen -t rsa
~/.ssh/id_rsa is your private key, ~/.ssh/id_rsa.pub is your public key
<localuser>@<localserver>$ chmod 700 ~/.ssh
<localuser>@<localserver>$ chmod 600 ~/.ssh/id_rsa
Transfer =<localpublickey>= to =<remoteserver>=.
<remoteuser>@<remoteserver>$cat <localpublickey> >> ~/.ssh/authorized_keys
<remoteuser>@<remoteserver>$ chmod 700 ~/.ssh
<remoteuser>@<remoteserver>$ chmod 600 ~/.ssh/authorized_keys
Now you should be able to login in to =<remoteserver>= as
=<remoteuser>= from
=<localuser>@<localserver>=. Perform these steps for each
pair of servers that need to synchronize with each other.
---+++ Git Installation
Install git on each server. For some linux distributions, git is not
in the standard repository. For example, for !CentOS, one has to
install the EPEL packages first.
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
To install Git from the source:
* Download the latest version of git (currently http://kernel.org/pub/software/scm/git/git-1.7.1.1.tar.gz)
* Uncompress download and cd to the installation directory.
* Use =./configure=, =make=, and =make install= to install git
Install git from binary distribution:
* Fedora/CentOS: =yum -y install git=
* Unbuntu/Debian: =apt-get install git=
After git has been installed, check the version using =git --version=. Ensure the version is at least 1.7
---++++ Distributed Mode
Perform these steps for all remote sites. <remotename> should be
different for each remote server.
cd ./twiki
git init
git config user.email “<localusermail>”
git config user.name “<localusername>”
git remote add <remotename><remoteuser>@<remoteserver>:<remoterepo>
---++++ Centralized Mode
Perform the following steps on the central server. <repository>
is an arbitrary directory where the central repository is located.
cd <repository>
git init --bare --shared
Perform the followikng steps in each (local and remote) TWiki
installation. <localname> should be different for each remote
server.
cd ./twiki
git init
git config user.email “<localusermail>”
git config user.name “<localusername>”
git remote add <centralname> <centraluser>@<centralserver>:<centralrepo>
---++++ Check installation
After these steps were done in all sites, you can test that these
configurations are working by executing
cd ./twiki
git fetch <centralname>
for centralized installation; use =<centralname>= for distributed installation. If no warning or error surfaces, continue.
Otherwise, check the configuration, as indicated by the warning or error.
You can get more information how to install git on http://git-scm.com/. If you
have difficulties authenticating through git and ssh, see [[#GitTips][below]].
---++++ Repository initialization
Create a file =.gitignore= to ignore files that should not be synchronized.
git status
will show all untracked files. Now create the =.gitignore= file from
these. A sample file has been provided in =lib/TWiki/Plugins/GitPlugin/dot.ignore=.
If you want to remove some file =<file name>= or directory
=<directory name>= from the repository,
append their path to the =.gitignore= file and then use the following
to remove them from the git repository:
git rm --cache <file name>
git rm -rf --cache <directory name>
git commit -m<comments>
---+++ Configuration Editor
* Open http://<localserver>/twiki/bin/configure in your browser.
* Select =Git= as the !StoreImpl mode in Store settings. If you can't select it, please view the notes below.
* Some items should be filled in the “General git settings” section.
* ={Git}{mergeMode}=: Select centralized or distributed correspoding to the intended mode of operation, matching the git installation.
* ={Git}{Path}=: The path of the git executable. Usually, the path is =/usr/bin/git=.
* ={Git}{SSHKey}=: Specify the path where the private key stored.
* ={Git}{remoteName}=: Comma-separated list of remote sites names (referred to as =<nickname>= during setup above.
* ={Git}{localName}=: Name of the local repository.
* ={Git}{backupReposname}=: Name of the central repository.
* ={Git}{outSync}=: Comma-separated list of webs which should not be synchronized and have not been added to the git repository (see the section "Selection of Webs not to be synchronized" below).
* ={Git}{root}=: The path to the git repository; must be the parent directory of the =data/= and =pub/= directories.
* ={Git}{LogPath}=: The path where the merge log file should be placed. (A good location may be =~apache/log= or ={Git}{root}/gitLog=.
* ={Git}{MasterEmail}=: if there are errors or warnings during a merge, GitPlugin will send emails to those listed.
---++++ Selection of Webs not to be synchronized
Any web listed in the ={Git}{outSync}= configuration parameter will
not be synchronized. These webs use RCS as the repository as in
standard TWiki. The list of webs excluded from synchronization should
be consistent between sites.
When you want to create an out-of-sync web
1 Add the name of the web to =.gitignore=
1 Add the name of the web to ={Git}{outSync}=
1 Create new web
When you want to change a synchronized web to an out-of-sync web
1 Add the web name to =.gitignore= file and ={Git}{outSync}=
1 =git rm -rf --cache <directory>= removes the web from the git repository. (remember to remove the web both in data/ and pub/)
When you want to change an out-of-sync web to be synchronized
1 Remove the web name from =.gitignore= file and ={Git}{outSync}=
1 =git add .=
1 =git commit -m"<comments>"=
Notes:
1 Topics and attachments in out-of-sync webs can only be moved to the out-sync webs and topics.
1 Topics and attachments in synchronized webs can only be moved to the synchronized webs and topics.
1 When deleting a topic or attachment from a synchronized web, it will be permanently deleted and not moved to the Trash web.
---+++ Repository preparation
To guarantee the local initial repository is based on the current
revision (to avoid conflicts which maybe not easy to be resolved) be
sure to follow these steps.
If this is the first TWiki installation to push to the server
1 Use =git add .= and =git commit -m"<comments>"= to initialize the local repository.
1 If distributed mode is used, do nothing.
1 If centralized mode is used, execute =git push <centralname> master:<localname>= to upload repository.
If this is a TWiki installation that is to be added to an existing
setup
1 The directories which will be synchronized should be moved out of the =twiki= directory.
1 If distributed mode is used, for each remote site
1 execute =git fetch <remotename>= to fetch the remote repository.
1 execute =git merge master/<remotename>=
1 If centralized mode is used
1 execute =git fetch <centralname>= to fetch the remote repository
1 execute =git merge <centralname>/<remotename>= for each remote site
1 Move local webs to by synchronized into the =twiki= directory
---+++ Synchronization Setup
For web-based synchronization, create a topic with content
=%GITMERGE{""}%=. Synchronization can be triggered either
manually, by viewing this topic, or automatically, by executing a
=cron= job.
When sychronization encounters conflicts, several buttons will be displayed with each file to allow remeding the situation:
|*Button*|*Function*|*File has been*|
| _Add_ | Adds the conflicting file to the repository | modified, added |
| _Delete_ | Deletes the file | deleted, added |
| _Retrieve_ | Checks out the latest version of this file from the repository | modfied, deleted |
| _Ignore_ | Adds the name of this file to =.gitignore= file | added |
To invoke synchronization via =cron=, edit the =crontab= file for the
=<localuser>= and append
*/1 * * * * cd /path/to/twiki/bin; ./view /TWiki.AutoMerge
where =/path/to/twiki/= is the path to the local TWiki installation,
and =AutoMerge= is the topic with the =GITMERGE= tag. The above entry
will perform synchronization every minute. Adjust the interval time as
needed.
---+++ Converting webs between out-of-sync and sync.
GitPlugin supports the conversion of a traditional TWiki web to a
synchronized web while preserving the revision history. Refer
to GitPluginConvert for more details.
---++ Synchronization of plugin work areas
If a plugin work area needs to be synchronized between sites, these can be selected in the =configure= script. For example, the repositories for the ActionTrackerPlugin or TagMePlugin benefit from being shared across sites.
Synchronized work areas will be stored in =gitWorking/= and symbolic links will be created to this area from =working/= to allow restoration of the original work area, if needed.
It is best to set this up ahead of time, but if the plugin has already been in use, the work area will be automatically moved. It may be necessary to remove the original working area from the git repository as git considers both the linked to file and the link different in the repository.
---++ Limitations
1 Ensure that the TWiki user can access the =ssh= private key.
1 Because the Plugin needs to modify the !LocalSite.cfg, if this file has already been created, =$TWiki::cfg{StoreImpl}= needs to be modified in the file =lib/LocalSite.cfg= directly. Alternatively, delete this file and revisit =configure=.
1 Conflicts caused by renaming or moving webs are not resolved. Consequentially, and ordinary user is not allowed to rename webs. Such must be performed by an administrator who should then update local repository with the latest content from the other sites and then perform the renaming.
1 After merging, the revision history may be changed.
1 The web configured as =$TWiki::cfg{Git}{LocalTrashWebName}= will serve as the Trash web for out-of-sync webs. The put back operation is not supported. This web should be added to =.gitignore=.
*WARNINGS*
%ICON{"warning"}% When synchronizing very large sets of changes the process may take larger than is permitted by the webserver and the update may be interrupted. Therefore, it is better to rely on cron for updates, or, when a large sychronization is anticipated, to perform this from the command line.
%ICON{"warning"}% git is somewhat sensitive to transfer interruptions, which might require manual completion of the push/fetch cycle.
---++ %TOPIC% Global Settings
Configuration for %TOPIC% are performed via =configure=. Ensure that all variables
defined for the %TOPIC% are given appropriate values, per their description.
---++ %TOPIC% Installation Instructions
* This plugin works only with TWiki 4 and beyond.
There are two ways to install the Plugin.
* Download the ZIP file from the Plugin web (see below)
* Unzip ==%TOPIC%.zip== in your twiki installation directory. Content:
| *File:* | *Description:* |
| ==data/TWiki/%TOPIC%.txt== | Plugin topic |
| ==data/TWiki/%TOPIC%Directories.txt== | relevant topic |
| ==data/TWiki/%TOPIC%Convert.txt== | relevant topic, convert outSync web to sync web |
| ==lib/TWiki/Plugins/GitPlugin.pm== | Plugin Perl module |
| ==bin/gitop== | Invokes the plugin. Must be executable by the TWiki user |
| ==bin/viewgitconflictfile== | Invokes the plugin. Must be executable by the TWiki user |
| ==lib/TWiki/Store/Git.pm== | Realizes git as the store module; interfaces with Store.pm |
| ==lib/TWiki/Plugins/GitPlugin/git.pl== | Runs the git commands. |
| ==lib/TWiki/Plugins/GitPlugin/GitAction.pm== | Realizes the confirmation button in a topic with conflicts and a selection button that enables determination which revision to keep. |
| ==lib/TWiki/Plugins/GitPlugin/GitRun.pm== | Similar to =Sandbox::sysCommand=; formats a command and submits for execution. |
| ==lib/TWiki/Plugins/GitPlugin/SyncWorkingDir.pm== | Realizes merging the work_area of plugin which need to be synced |
| ==lib/TWiki/Plugins/GitPlugin/View.pm== | Similar to =UI::View=; displays different versions of conflicting attachments |
The other is as follows.
* Download the tgz file and installer file from the Plugin web (see below)
* Put the the two files with your bin, lib, data and pub directories together.
* run =perl !GitPlugin _installer= to install !GitPlugin automatically.
---++ Some tips for using =git=
#GitTips
If you have trouble authenticating to the central repository, try the following:
* Create a shell script as follows:
#!/bin/sh
ssh -i /path/to/private/key "$@"
* set the =GIT_SSH= environment variable to point to this script.
---++ Plugin Info
| Plugin Author: | | Proposed and initially developed as class project for TWiki:Main/ThomasWeigert 's CS397 "Software Systems Development" at Missouri University of Science and Technology, Fall 2009 by Joshua Bohde, Afik Cohen, Peter Hoeft, Jeff Arneson, Jessica Egler, Kenneth Perry, Steven Wallace, and further enhanced by Yao Zhou and Brute Ma of !UniqueSoft LLC. |  |
|
| License: | GPL ([[http://www.gnu.org/copyleft/gpl.html][GNU General Public License]]) |
| Plugin Version: | 10711 |
| Change History: | |
| 01 Nov 2011 | Various performance improvements to avoid obtaining full git log, which is extremely slow in very large directories. |
| 11 Jul 2011 | Add ssh port option |
| 30 May 2011 | Display untracked files correctly. |
| ^ | Modify the format of error message when there are errors/warnings in AutoMerge |
| 16 May 2011 | Support to convert outSync web to sync web |
| ^ | Add a switch to control error email |
| 29 Apr 2011 | Fix the bug that When a file is found to conflict or be added which has a file name that includes a # sign, then this character is omitted in the displayed file name. |
| ^ | The comment of attachment which commit in AutoMerge will be retained |
| 21 Apr 2011 | Add $VERSION and $RELEASE information. |
| ^ | Enable file name can use non-word. |
| ^ | Enhancement that work area of ActionTrackerPlugin can be synced. |
| 22 Mar 2011 | Fix the bug that if creat a web based on a no attachment web, it will prompt error when attach |
| 18 Mar 2011 | For untracked files, Git Plugin needs to recognize other files than just the files under data/ and pub/ |
| 25 Feb 2011 | Enhancement that while there is warnings/errors during sync, emails will be sent to who care it. |
| ^ | Sync log will be recorded. |
| ^ | When topic has merge conflict, email will be sent to the author who last modified it. |
| ^ | Enhancement that work area of TagMePlugin can be synced. |
| 01 Feb 2011: | Fix the bug that When choosing a particular item to merge ("Select" button) where the conflict is in a table, the replacement breaks the table by inserting empty lines before and after the merged text. |
| ^ | Enhancement AutoMerge page that have a checkbox that selects all checkable items when there are files detected not under revision control. |
| ^ | Enhancement conflict topic that allow users to select conflict attachement and meta data.( 7 types: parent, allowtopicview, denytopicview, allowtopicrename, denytopicrenmae, allowtopicchange, denytopicchange ) and add anchor of conflict content. |
| 11 Jan 2011: | fix the bug that pressing "add" button of files in pub which edited out of twiki is not correct. |
| 22 Nov 2010: | Added support for synchronizing edits performed outside of TWiki. |
| ^ | leverage !RcsFile and !RcsWrap/RcsLite to manage non-sychronized webs. |
| 01 May 2010: | Initial release |
| TWiki Dependency: | $TWiki::Plugins::VERSION 1.1 |
| CPAN Dependencies: | none |
| Other Dependencies: | Git 1.7 |
| Perl Version: | 5.005 |
| Plugin Home: | http://TWiki.org/cgi-bin/view/Plugins/%TOPIC% |
| Feedback: | http://TWiki.org/cgi-bin/view/Plugins/%TOPIC%Dev |
| Appraisal: | http://TWiki.org/cgi-bin/view/Plugins/%TOPIC%Appraisal |
__Related Topics:__ %TWIKIWEB%.TWikiPlugins, %TWIKIWEB%.DeveloperDocumentationCategory, %TWIKIWEB%.AdminDocumentationCategory, %TWIKIWEB%.TWikiPreferences, %TWIKIWEB%.%TOPIC%Directories
%META:FILEATTACHMENT{name="DistributedMode.jpg" attachment="DistributedMode.jpg" attr="" comment="Saved by install script" date="1300761712" size="73824" stream="GLOB(0xbcd6d00)" tmpFilename="/tmp/x7ujRZSn4V" user="BaseUserMapping_333" version="1"}%
%META:FILEATTACHMENT{name="CentralizedMode.jpg" attachment="CentralizedMode.jpg" attr="" comment="Saved by install script" date="1300761712" size="87148" stream="GLOB(0xbcd6460)" tmpFilename="/tmp/7VNZyt9LKy" user="BaseUserMapping_333" version="1"}%
%META:FILEATTACHMENT{name="select_all.js" attachment="select_all.js" attr="" comment="Saved by install script" date="1300761712" size="296" stream="GLOB(0xbcd6c90)" tmpFilename="/tmp/xxGRarZ158" user="BaseUserMapping_333" version="1"}%
%META:TOPICMOVED{by="BaseUserMapping_333" date="1278036093" from="Sandbox.GitPlugin" to="TWiki.GitPlugin"}%
%META:FILEATTACHMENT{name="Logo-Green-web_w.jpg" attachment="Logo-Green-web_w.jpg" attr="" comment="Saved by install script" date="1305540952" size="11494" stream="GLOB(0xaa64b58)" tmpFilename="/tmp/AFa4qkNCA8" user="BaseUserMapping_333" version="1"}%