NOTE: the script this topic is about has been merged with core twiki and is in CairoRelease. The current topic for discussing it is UpgradeTWiki. This topic needs to be refactored out of existence. -- MattWilkie - 01 Sep 2004

I just had a realisation that we all should have seen ages ago!

TWikiTopics are Versioned. which means that all i need to do to upgrade from one version of the TWikiTopics to another, is to generate a PatchFile of the changes that have been made to my old installation (ie rcsdiff -d "the old release date") and apply that patch to the new TWikiTopics.

To make this easier, i think we should store the date / tag the TWikiTopics in a distributed Topic file that should not get edited by the user. (see WikiDistributionVariable)

The point of this script is to make it simpler to upgrade the distributed topics (with local customisations) to a new release (I track the Beta's at work). It is not intended to consider changes in the structure of our releases (until after that happens).

-- SvenDowideit - 07 Jan 2004

Would this merge changes where installed topic and the distributed topic have been modified differently?

• YES - that is the point of the script.

i.e. both have a WebPreferences topic that was the same at version 1.4 but installed topic is now 1.6 with some changes made to standard variables and some custom variables added and the distributed topic now has a version 1.9 with new variables and all the old defaults.

what the script does

foreach file in the oldInstallDataDir {

• look for the highest common revision with the newReleaseFile
• if there is no common revision copy the oldFile
• if the common revision is the last on in the old file, copy the newReleaseFile
• else,
  • do a diff between the higest common revision and the head of the old file
  • copy the newReleaseFile
  • patch it with the created diff (rejectes are saved)

}

foreach file in the newReleaseDir

• look to see if it exists in the newDestination
  • if not, copy it

}

Note:

• if there is no rcs file during the either loop I just copy that file.
• I also look for files from the _default dir in each user-created web
• the user then needs to eyeball the *.rej files
  • TWikiPreferences had trouble only with the COPYRIGHT patch
  • the only other problem rejects were WebHomeTopics that changed too dramatically (I may have created them by hand too)

still todo

• create rcs files when they are missing
• after patching the script should commit the change (its not in the rcs file yet)
• we can improve the chances of finding upgrades by doing more diffs

Questions/Discussion

What happens if a site:

• Solved the lock ownership of pages from nobody to whatever their server uses by deleting all the history files ? eg the few hundred wikis installed using the TWikiUnixInstaller ? (Not counting forks)

  Even before an installer existed this was my personal preference for solving this issue, and comments on IRC, the Support web & elsewhere show this is a fairly popular method. (Since it's quick, simple and works)

OR are you expecting to run this script on TWiki.org (say) before a release to produce a patch which people are then expected to apply to their Wikis locally? ie it doesn't matter if they have removed their local version info.

• Reading into this in more detail I suspect you're expecting to supply to users a patch file to upgrade their web text. Correct?

-- MS - 12 Jan 2004

I honestly did not expect anyone to be so stupid as to delete the ,v files. Its trivial to change the lock owner, and in fact all i do is to remove the lock itself. basically, this sucks

I was intending, and as my main focus am still on generating a diff based on a diff on the user's currently installed ,rcs file and then applying that to the new topic, but for installations that have the braindead case you mention..... wait..

so long as you know what the dat of your existing install topics are, we can use that at the higestCommonRevision, do the diff to that (in the new topics) and then move on. I did not do this as I know that my install is a mix of different versions as I added topics that were new and needed for new features..

-- SvenDowideit - 12 Jan 2004

At work I do sometimes incremental updates and copy&paste the content of a TWiki.org topic over an existing topic at work. That means the revisions numbers and dates certainly gets out of sync.

The only reliable way to upgrade an existing topic is to check in the latest version from the installation package. Which should be simple to implement.

To AutomateTWikiDocUpdates, we could distinguish between different types of actions taken on upgrade, depending on the topic.

-- PeterThoeny - 12 Jan 2004

Yeah, I've done the cut and paste too, notably to add the WebRSS topics - but the script seems to have picked that up (somehow) and just replaced that with the new version (i think... need to check).

my intended strategy for this situation (not implemented yet) was to find the highest common revision, then from there do diffs between the remaining revisions in the existing file against all the remaining revisions of the new topic - and if there is a match, assume that you want to upgrade to the new topic.

-- SvenDowideit - 13 Jan 2004

Wouldn't that be simply compare the top rev of the installed topic with the top rev of the package (minus some meta data like META:TOPICINFO) and update if different? See also other scenarios in AutomateTWikiDocUpdates.

-- PeterThoeny - 13 Jan 2004

> I honestly did not expect anyone to be so stupid as to delete the ,v files.

I've done this twice now. The first time was to shoehorn TWiki into a host that only allowed 5mb. It worked, but I ended up deinstalling for other reasons (couldn't get authorization to work). The second time was to quickly remove the topic locks as MS described.

> Its trivial to change the lock owner, and in fact all i do is to remove the lock itself

which is more trivial:

• select "*,v" and slap delete (or rm */*,v if you're a unix head),
• open a command prompt, run perl -ie 's/...regexp that can't be remembered and must be looked up.../g'. Of course to be really correct, you need to grep the regexp first to make sure it only changes the lock record and not topic text (the regexp example in the installation docs is not restrictive enough). now fixed, the relocking script now only checks the first 10 lines which is before the topic text starts. there still might be room tighten the regex.

Also note that there is a feature request to delete topic revisions older than xxxx.

Don't construe this as a diatribe against your approach. I'm not knowledgeable enough to know. Truth to tell I don't care how it's done as long as it works!

-- MattWilkie - 13 Jan 2004

Here is another requirement that we have at our site, and I think this is related.

• The process group maintains templates for specific requirements. The projects make sure all the required information is made available in a topic instantiated with that template.
• Once in a while, the template is updated with new sections. When that happens, all the projects have to make sure that all the previous instantiated topics get updated with new sections. (This is a manual task as of now.)

With the "Topic upgrade" concept, clicking on "upgrade to new template" should result in incorporating new sections automatically. It may not always be automatic; in which case, it can only show the differences. (Assumes that topics know which template they instantiated from.)

-- VinodKulkarni - 14 Jan 2004

Sven,

> I honestly did not expect anyone to be so stupid as to delete the ,v files.

It's the simplest approach to solving the lock files problem.

• See also Search: (remove|delete).*(\*,v|,v|lock|RCS) - its not a small list

> Its trivial to change the lock owner, and in fact all i do is to remove the lock itself. basically, this sucks

It might be trivial, but as Matt says, rm is a tried and trusted tool for this. Also the TWikiInstallationGuide says to do do one of the following:

• unlock, and relock all the files by hand
• delete the files. (The simplest thing that can possibly work)
• Or run a perl oneliner over all the content - which incidentally can (and IME does) also change topic contents as well as the locks.

I suspect that the majority of people who look at this will go why cause this problem in the first place? I know I've thought that more than once.

On a more positive note, I think one thing you might have missed though is this comment I made above:

• Reading into this in more detail I suspect you're expecting to supply to users a patch file to upgrade their web text. Correct?

If you hadn't considered it - please mull it over - I thought of it whilst trying to think how your approach might work (since I couldn't see how that it could and know that one of the most popular approaches to the lockfiles is just to delete the history). Whilst I thought your approach was fragile, if you do diffs of the content between versions you distribute and then distribute only patches to people to upgrade their topics, this is closer to the idea that people's sites act as CVS clients. (albeit a "bad" reimplementation of CVS, but that's a different issue) I'm not convinced this approach will work for certain either, but I suspect it's got a greater chance of success

Creation of patches based on TWiki.org content, and distribution and then patching is more realistic IMO that the approach you were suggesting. (Large amounts of software is upgraded on a patch basis rather than large tar basis afterall)

-- MS - 14 Jan 2004

Perhaps if RcsNonStrictLocking was fixed the lock part of the issue would just go away.

-- MattWilkie - 14 Jan 2004

no. you cannot create a patch file that goes from an unknown version to the current one, which is what I am trying to do with this script. I am currently patching the newRelease topics with the changes made by the user on their versions. I will investigate patching the user's version with the changes from the detected highestCommonRevision to the newRelease, but I suspect that we are talking about symetrical merge cases

which is better?

1. patching the newReleaseTopic with the user changes ensures that the next upgrade is as simple as possible (and we are talking about distributed topics here)
2. patching the users version with the changes from highest common to newRelease keeps the users revision files, but causes the next upgrade to pick the wrong highestCommonRevision.

which is why i wrote the script the way I did. Maximum flexibility, and repeatability.

I'm with Matt - is there any reason why the distributed files are locked? couldn't we unlock them before we package the file?

-- SvenDowideit - 15 Jan 2004

Why replace the user's revision history for the file? How do you detect "highest common revision?" If you check the actual content (i.e. via some hashing algorithm), then you could easily keep the user's history and get "repeatable" highest-common-version selection.

Each update would do the following:

1. Check in the new version of the file to the user's local RCS, without any modifications.
2. Check it in again, this time patched with all of the modifications made since the last update.

Just patch the "current" topic text with the merge results and RCS add it as a new revision onto the version history. Then all the user has to do if they don't think the merge went well is look at the TWiki diff between the updated revision and the old one (now 2 revisions back) and bring forward anything that got munged. Of course the .rej files should also be produced. Perhaps we could put together some kind of interface for locating, presenting, and allowing the user to deal with these?

Re: deleting the ,v files. We need to fix the locking issue and take that recommendation out of the installation instructions immediately (where is it? I don't remember reading any such recommendation when I first installed TWiki). I'm not really all that familiar with working directly with RCS, but there doesn't seem to be any good reason to require the user to do this kind of thing rather than take care of the required changes in the distribution somehow. I don't like the idea of an installer that deletes the histories...if you're going to write an installer, why not write one that actually has a more robust lock-change method than that broad one-liner in the docs?

-- WalterMundt - 15 Jan 2004

> if you're going to write an installer, why not write one that actually has a more robust lock-change method than that broad one-liner in the docs?

I discussed this* on the TWikiUnixInstaller pages - and I still maintain that deleting the histories is a robust lock-change method at installation time. It's fast, efficient, and simple. A better solution for admins is to ship without the histories at all since they are largely irrelevant to users. Shipping with histories causes problems for admins and

• * See here, and here

The pre-local-install histories are by and large irrelevant to a user. The admin don't care either. It's only now when it's started being used, and it's not something you can rely on existing anyway. This has been the simplest way of changing the lock for the past 4 years, and as a result the fact it exists, an installer has done this for the past 2-3 years and a better installer has done it for around 7 months.

Bear in mind you need to reliably detect the web users on a system that might not be running your personal preference of a server, not necessarily using your OS, and hence have lots of things in different places that you're not expecting and then you begin to realise the issues that you're facing. Contrast that with the solid, simple, reliable cd data; rm /,v .

Someone who hasn't appeared to touch a single line of code in months isn't in a position to criticise a method that does work IMHO. Write the code to implement the change and I'll add it into the installer - or you can - it's not as if the code is hidden. Otherwise, I'm not going to bother - this method is simple & robust and portable between webservers & Unix systems.

It only deletes the histories at install time - a time when a system does not have a history .

No other product I've ever come across has ever relied on shipping their entire version control system* with their product just to deal with a simple issue like upgrades. It's overkill, and a very fragile method.

If you've ever worked as a systems admin the one thing you will realise is this:

• Stuff that is easy to install gets installed, unless there are huge benefits in using the thing that's difficult to install.
• If an upgrade is ever needed and it's a real PITA, it's worse than never using the product in the first place, again unless there are huge benefits in using the thing.

Ask yourself why Stata decided to ditch TWiki when it moved from an engineer playing to the IT dept managing it? Was it because socialtext did a better job in terms of features & acceptance? To be honest I doubt that was the real reason - I suspect it was all about managability. Likewise I've heard of many people ditch TWiki (including, I believe, TWiki core team members...) when it's become to awkward to manage code/upgrades/whatever.

If TWiki is to have corporate general, widespread acceptance it requires as little work to be done by a systems admin as possible - and this includes doing the job of the packager (resolving conflicts, and changing lock files is not the job of a systems admin), this also includes the ability for users to skin their own work without root access on the server, and so on. That is why I've worked on an installer*, and that's why I've pushed for all of these areas including the ability to simplify backups of local data through data and code separation .

• * Why else would I bother? If I can write an installer, I clearly don't need it myself most of the time.

The topic of this page is a nice, elegant and wrong solution to patch people's systems in this way - you'll get rejections and conflicts, with the result people look to migrate their content away from TWiki, or worse run with insecure systems as time goes on.

• Might be worth moving the bulk of this rant to RelockingRCSFiles

-- MS - 15 Jan 2004

It's worth noting that this is a special case of read write offline wiki

-- MS - 23 Jan 2004

Corrected no longer true statement about the one-liner re-locking script being too agressive. It now only checks the first 10 lines which is before the topic text starts. there still might be room tighten the regex for edge cases.

-- MattWilkie - 24 Mar 2004

---

How is this script used? Does it need to be edited first? Does it have to be run on the server or can it be run on downloaded local mirror which is then uploaded? Where are the docs man?

-- MattWilkie - 29 Mar 2004

Originally I wrote here "what problem is this script solving", but I think I see that now.

Like another person above, I'm suspicious of the need to distribute the entire revision history of a new version along with the version in order to make it instalable.

I'm also suspicious of the idea that changes made to TWiki topics under the previous version would necessarily be relevant in the new version... the idea that a mechanical merge will solve more problems than it creates.

There are a few files that will definitely be edited for specific purposes under the old version, where the edit information is needed: WebPreferences and user information. Could these two (at least) be better solved from inside the Twiki engine, where knowledge of the meaning of the content of these resides?

At the point of upgrade, its a fact that the users and variables should be inherited into the new version from the old: these could be done intelligently.

It may be the case that where other Twiki distributed topics have changed, the best thing that an upgrade script could do is just tell the admin: hey dude, your users changed this list of topics, and you're going to have to see why.

What are the kinds of changes that people make locally in real life to topics that come with the distribution?

In my case, it's only the front page... all other local changs are in a local web...

-- MartinGregory - 01 Apr 2004

Matt - I'm not sure anymore when i next have to upgrade my work TWikiTopics I'll get back to it)

Martin - you'd be right, except that historically there are configuration changes that are littered all about the TWikiWeb, MainWeb and _defaultWeb. the example that springs to mind are the WebSearch Topics. At my work we have a few small customisations to these, and doing a cvs update style upgrade works beautifully. I just used the versioning information because as i wrote at the top, because it meant that I could automate what i used to do laboriously by hand. The final outcome was I used the same amount of time to write a script as i took me to upgrade once by hand.

Its not that it is necessary to use the versioning to do the upgrade, it just simplifies the task dramatically. In exactly the same way as doing team based development is made simpler by useing a versioning system.

(as a bonus challenge - could someone please explain to me why you would not use a tool that was available (the versioning info), rather than tracking and updating the topics (the ones that merge cleanly) using that tool? I am having trouble understanding your reluctance)

-- SvenDowideit - 01 Apr 2004

Because you deleted all the ,v files to free the locks, as per the docs?

-- MartinGregory - 01 Apr 2004

anything else? because if thats all, we can still use the ,v files in the new release package to automate most of the topic upgrade - I was just really surprised by it - and even more surprised that it was the documented way to do it. I'd have written a script to free the locks instead (if i'd have known) (as i don't like to loose information needlessly) (i worked in analysing weather data for a while at uni, and if we threw out dodgy data, we wouldn't know anything about weather trends in Australia )

-- SvenDowideit - 01 Apr 2004

Martin - I was responding to your statement:

"Like another person above, I'm suspicious of the need to distribute the entire revision history of a new version along with the version in order to make it instalable"

and I still cannot see what your suspicion is based upon (and even worse, what the point of having that suspicion is). It makes the work of the script one heck of a lot simpler, and personally i don't see any reason not to do it.

It appears that I still get annoyed with the number of people that seem to think that my approach to upgrading my twiki topics is bad, but that they seem unwilling to write some code themselves (ask quite honestly this script does exactly what I needed to upgrade my twiki, using all the versioning information (both in the distributed topics, and my modified topics).

so stop theorising, and write some code already!, or if you want me to change something, enthuse me, make me understand what you want, and why its going to be better that way.

(I was looking forward to having a conceptual codeing argument with MS, but somehow he got banned from twiki.org, so that didn't work out )

PS: Matt - this does not apply to you I'll docco what I have soon (but i am more worried that to upgrade your busted rcs files we will need to do something different)

-- SvenDowideit - 02 Apr 2004

I accept that my objection is theoretical and aesthetic.

I've come to see that getting to a position where code, configuration and data are cleanly separated will not be a quick process. So...

If the do-it-based-on-version-info system can really be made to work smoothly and easily, let's do it.

-- MartinGregory - 02 May 2004

> Its trivial to change the lock owner, and in fact all i do is to remove the lock itself. [sayeth sven]

How, exactly, does one do this? I too would prefer not having to futz around with changing the owner if it's not required.

And where is the "build twiki release script" so that this can be done before distribution?

-- MattWilkie - 27 Jun 2004

Matt, regarding your question on changing the lock owner...

• To remove a lock on an RCS file, use the command rcs -u[rev]. Note that this sends an email to the locker; a script should generate that message, otherwise you will be interrupted. The messagge is terminated by "." or EOF. In some versions of RCS you are asked whether you want to break the lock, and it requires to answer "y".
• To lock an RCS file, use the command rcs -l[rev].
• Note that the lock is indicated in the RCS file by a line containing the single word "locks". On the next lines you have each lock in the form of a TAB character, followed by the login name, followed by ":", followed by the revision number, followed by ";" followed by " strict;" if the locking is strict. So you could also edit this file directly in a script.

-- ThomasWeigert - 27 Jun 2004

Thank you Thomas! I've used your info to put something together at RelockingRCSFiles.

-- MattWilkie - 28 Jun 2004

I have put the contents of this script into a package (UpdateTopics.pm), tidied it up a bit, and included it in UpgradeTWiki.

(note that Martin's 'tidying' might be other people's 'defacing' or 'buggifying'. Caveat emptor )

-- MartinGregory - 18 Jul 2004

Added _NOTE: the script this topic is about has been merged with core twiki and is in CairoRelease. The current topic for discussing it is UpgradeTWiki. This topic needs to be refactored out of existence. _ to top. The TopicClassification indicates this is still a work in progress, an oversight?

-- MattWilkie - 01 Sep 2004

---

CategoryUpgradeStrategy