TWikiWithIntermittentConnectivity < Codev

TWiki with (very!) intermittent connectivity

the problem

imagine you have users with a laptop going to non connected areas as a archive, a church or whatever your imagination can construct, maybe in some medieaval institution in middle Italy. Imagine that the data they use, produce, update is kept on a central TWiki server, which we want to keep synchronized with the many travelling laptops. imagine further that the duration of being off-line is quite long, maybe a whole day and imagine that our users are really non technical people.

...

it looked like a nice problem so we tried to face and solve it. we did not come much farther than this notes. maybe can be of some help for the rest of the community.

a proposal

the following text has been machine-translated. please feel free to correct it. the original Italian text can be found at rev 1.1

receiving modifications carried out while offline

there are some persons who separate themselves from the net and would want to remain active on our central twiki. the idea is to keep an apache server active on the separabile machine, with an entire copy, in reading-writing, of the central twiki site.

it must be possible to integrate on the situated one centers them the local modifications and on mirror the modifications it centers them. the realized mechanism must be simple to use - attivabile with a commando from twiki same and must reduce lessened the communication gives to you on the net. TWiki could quite show one page type: "we have connnettivit� hour, are more than x hours that we do not bring up to date ourselves, I would prefer not to proceed".

what to do

a method would be to store at every contact realized the version of every topic found on the central server, to "rcsdiff - u" on the peripheral site and to "patch" on the central site.

we call 'isola' the computer that separates periodically from the net and therefore from the TWiki central server. we call 'continente' the computer on which we maintain our situated TWiki centers them. at the moment the only island is edelweiss and the continent is berken.

how to do it

to the conclusion with succeeding of a modernization, for every web followed, memorizzano the names and the versions of the received pages. the commando comes given on the island, at this point dawned, and is:

cd /home/web/twiki/data
for WEB in Ibo Main Private
do for FILENAME in $WEB/*.txt
   do echo `basename $FILENAME`:`rlog -r $FILENAME | grep ^revision | tr -d -c "[0-9.]"` 
   done > $WEB/lastcontact.log
done

the output will be copied on .

delivery

to the beginning of a modernization, debit newly the same commando, writing the output on sending.log.

to this point, for every rows for which the version he differs, it must generate rows diff to give in meal to patch on the continent all the differences refused (the reject) go given back to the sender and notified to a responsible.

the following commando generates a list of names follows itself from the version number, the list comes read two lines for time, the uneven lines refers to the last received version, those pars to that one produced.

diff - u lastcontact.log sending.log|grep "^[-+][^-+]"|cut - b2-

the output of this commando it can be used for a small program awk, like in following ' oneliner ':

cd /home/web/twiki/data
for WEB in Ibo Main Private
do diff -u $WEB/lastcontact.log $WEB/sending.log | 
   grep "^[-+][^-+]" | 
   cut -b2- | 
   sort | 
   awk 'NR%2{
              split($0,tmp,":");
              last=tmp[2];
            }
!(NR%2){
         split($0,tmp,":"); 
         name=tmp[1];
         current=tmp[2];
         cmd=sprintf("rcsdiff -u -r%s '$WEB'/%s",last,name); 
         system(cmd);
       }' ; done &> sending.diff

to this point it must copiare/inviare towards the continent the several one update.diff generates you for every web. then it must put itself in conditions of being able to give the commando patch - p0 < update.diff.

the problem is in the system of locking RCS used from TWiki. fundamentalally, in order not to have cottages, must operate as the same customer under which it turns the serveur web, typically nobody. in this case it will be enough to then apply the patch generated for every web and, for every rows modified, to give the commando us - l (checkin + checkout + lock). TWiki would not have to notice of null, indeed, it notices of therefore little that it does not send not even the notifications.

probably the method simpler in order to carry out all that, going around also the problem of the customer, is to write a small cgi-script to which one by one sending or all with the rows update.diff. this cgi-script would be very well to the side of the several one view, edit, save etc etc that makes part of TWiki. ago a POST of the name of the web to correct and the rows diff. the script, that we could also call patchweb, applies the patch to the web enclosing to every topic corrected the eventual ones rej it meets to you.

reception

once the modifications carried out on the island delivered, must demand the modifications happened on the continent. creed that this would go made outside from TWiki. even a simple one oneliner that copy from the web you follow all the rows modified after the last contact. not creed that is the case to lose time to carry out the same cycle diff-patch also to the inverse one, is problems connects to you with the numbers of the versions, does not want not to think to us.

ah, we do not forget ourselves about encloses to you, that they go copied also they.

they said that on the island the ridenominazione of the topic must be disabilitata, like pure me seems that we could render stricter TWiki regarding the acceptance of topic with names that are not WikiName .

like automating it

it is obvious that it will not be able to be demanded to a any customer to make all the steps over described here. the installatore of the situated premises will have to create the rows lastcontact.logthen it will be the system, as a result of a commando of the customer, to take care itself to generate the rows diffto carry out the contact, to pass i diff to the situated one twiki they centers, to receive the modifications she consequently centers them and to modernize the rows lastcontact.log.

hour I describe in scattered order what serves to me in a hypothetical and still nonexistent DiffPatchPlugin ? that it will be composed of two members, one serving installed on the continent and one customer installed on the several islands.

they said that in the situated one they isolate we will have to indicate to TWiki which are the web to consider of the mirror of the situated continental. this can be made defining one variable, even %DIFFPATCH_MIRRORINGWEB%. the web it does not list to you will be considers you of the all svincolati ones from those continentals. some islands will not want to receive the pages or the attachment if these exceed the dimension indicated in the variable one %DIFFPATCH_SIZELIMIT%. possibly on the continent we will be able to indicate which web can be replied on the islands and/or which not of the type %DIFFPATCH_WEBALLOW% and %DIFFPATCH_WEBDENY%. the values of default will be respective ' to and ' ninth '. the island will know qual' is its continent consulting the variable one %DIFFPATCH_REMOTESCRIPTURL%.

TWiki with (very!) intermittent connectivity

the problem

...

it looked like a nice problem so we tried to face and solve it. we did not come much farther than this notes. maybe can be of some help for the rest of the community.

a proposal

the following text has been machine-translated. please feel free to correct it. the original Italian text can be found at rev 1.1

receiving modifications carried out while offline

what to do

a method would be to store at every contact realized the version of every topic found on the central server, to "rcsdiff - u" on the peripheral site and to "patch" on the central site.

how to do it

cd /home/web/twiki/data
for WEB in Ibo Main Private
do for FILENAME in $WEB/*.txt
   do echo `basename $FILENAME`:`rlog -r $FILENAME | grep ^revision | tr -d -c "[0-9.]"` 
   done > $WEB/lastcontact.log
done

the output will be copied on .

delivery

to the beginning of a modernization, debit newly the same commando, writing the output on sending.log.

diff - u lastcontact.log sending.log|grep "^[-+][^-+]"|cut - b2-

the output of this commando it can be used for a small program awk, like in following ' oneliner ':

cd /home/web/twiki/data
for WEB in Ibo Main Private
do diff -u $WEB/lastcontact.log $WEB/sending.log | 
   grep "^[-+][^-+]" | 
   cut -b2- | 
   sort | 
   awk 'NR%2{
              split($0,tmp,":");
              last=tmp[2];
            }
!(NR%2){
         split($0,tmp,":"); 
         name=tmp[1];
         current=tmp[2];
         cmd=sprintf("rcsdiff -u -r%s '$WEB'/%s",last,name); 
         system(cmd);
       }' ; done &> sending.diff

reception

ah, we do not forget ourselves about encloses to you, that they go copied also they.

like automating it

there will be a TWiki topic on the satellite system which will offer the user the possibility to perform the contact. the user will be responsible of having established the net connection and he will have verified that the continent is available for receiving the data. on the other hand, if either are not available, it will not be that hard to construct some standard error message what will not be too surprising for the user either.

the page will show a form with a directory of all mirrored webs and will offer to the user the possibility to select a subset for the actualization. the action of the form will be a program that will generate all the diff files and post them to %DIFFPATCH_REMOTESCRIPTURL%/patchweb.pl on the central server. after generating the diffs and posting them to the central server the local program will redirect the browser to a remote page which will show the results (conflicts and new versions to be received). the remote patchweb will in turn conclude its action carrying out a post towards the satellite's patchweb, passing all the files and the most recent attachments. the interaction is concluded with regeneration of the rows lastcontact.log.

original version:

-- MarioFrasca - 10 Nov 2003

donated:

-- MarioFrasca - 09 Mar 2005

Or just visit http://translate.google.com/translate?u=http%3A%2F%2Ftwiki.org%2Fcgi-bin%2Fview%2FCodev%2FTWikiWithIntermittentConnectivity&langpair=it%7Cen&hl=en&safe=off&ie=UTF-8&oe=UTF-8&prev=%2Flanguage_tools

-- CrawfordCurrie - 09 Mar 2005

well, this automatic translator does not do such a bad job. at least the first paragraph is readable to me (knowing the subject and the original language!) now I must recover the format and I'm curious about the reactions on the content. then I will produce a real translation if I see that there is any interest around the problem.

-- MarioFrasca - 09 Mar 2005

I haven't given it much thought, but it occurs to me that the ReleaseLocksOnSave change in DevelopBranch makes offline TWiki much easier. You can call the save script, with the appropriate originalrev parameter, for every topic that was saved which you were offline, and it should merge the changes.

You would probably want the sync to operate in different modes - for example, "overwrite", "merge", and "interactive". Possibly even in different modes for different topics. Look at Windows briefcase (or probably better to look at ClearCase Attache) to see how others have solved this.

BasicForm
TopicClassification	Select one...
TopicSummary
InterestedParties
RelatedTopics	ReadWriteOfflineWiki