Hi Martin, I don't have anything concrete to add at this moment; I just want to (strongly!) encourage this discussion.

From my perspective the biggest problem is that there is no clean separation between local and as-shipped settings. The stumbling block to changing this seems to be concerns over backwards compatibility. Personally I would rather have a painful compatiblity-breaking upgrade once, and then easy "unzip and go" upgrades there-after rather than laborious upgrades every so many months (or years).

-- MattWilkie - 10 Apr 2004 - 14:26

This is an absolutely essential point to resolve. I've seen at least a couple of discussions in topics that were making good progress come unstuck when someone raises backward compatibility. Let me run this up the flagpole and see if anyone salutes:

"The Roadmap for implementation of a decent upgrade strategy will include a single well-thought out place where backward compatibility with nuisance features will be broken.  Users will be transitioned across this break with helpful once-off tools"

-- MartinGregory - 11 Apr 2004

Martin - yes, I'm eager to back anything that improves our ability to move forward. This also includes removing legacy code from the core when possible to reduce our maintanence cost for little used features

-- SvenDowideit - 11 Apr 2004

Martin, good that you take this initiative, I am loking forward to improvements in this area.

Easier installation and easier upgrade are very important steps to make TWiki more userfriendly. I see these two go hand in hand. This is a non trivial task since TWiki has a low requirement on server, e.g. the install/upgrade should be as generic as possible across different operating systems and user privileges (e.g no root access on IT managed server or hosted server).

Some thoughs on the install/upgrade script:

• Since Perl is already required (by definition ), install/upgrade script could be Perl based so that 3rd party installs are not necessary
• Make script interactive, e.g. prompts with remembered default values (so that next upgrade knows what to do)
• Do not require a compiler
• Do intelligent merge of preferences pages (site-level, web-level, user-level, Plugins settings). That is:
  • use content of new version
  • carry over old Set values
  • carry over user defined settings
• For existing topics, preserve existing topic history; create new revision with latest content
• PreinstalledTWikiPlugins need to be updated as well, including preferences settings; the number of Plugins increases with each release
  • Those Plugins do not require additional installs and do not require a compiler
• For non-root priviliged users possibly require to run two scripts:
  1. shell script, run as the login user (for bin, lib and templates)
  2. secured cgi script, run as the web server user (for the topics, e.g. data and pub)
• Best to have a data driven install/upgrade, e.g. no need to change the script. Each release could have configuration file(s) containing the details needed for the install/upgrade.
• The install/upgrade could pull the ZIP from TWiki.org, but a self contained setup should be possible as well. Do not assume that all corporations have internet access, e.g. some aerospace & defense companies do not even have firewalled access to the Internet.
• Possibly leverage some goodies from the testenv script

On backward compatibility, the TWikiMission states to "protect corporate investment (topic contents) from data corruption and incompatible changes". I do not see an issue if a few configuration pages change, however topic content in general should be protected from incompatible changes (for example, at my workplace we have over 35K pages).

-- PeterThoeny - 12 Apr 2004

If a significant structural change is needed to get TWiki into better shape, I would see the answer to the "protect investment" issue as "provide upgrade scripts".

-- MartinGregory - 02 May 2004

Some time ago we made the decision not to introduce upgrade scripts. The current scripts take care of TWiki topic format upgrades, e.g. from older attachment table format to the current format. That is, it is possible to view and edit older formats. The decision was made because a company might need to restore content from an old backup or want to merge content from different TWiki installations. If we provide an upgrade script it might be OK for the current version, but what if it is 3 generations later? (An earlier M$ Word could not read older Word file formats. This has been fixed after some outcries...)

An easy to handle upgrade/install script is not trivial but also not unsolvable.

-- PeterThoeny - 04 May 2004

This is an open source project. The older uprade scripts will always be available to those who might need to upgrade through a generation or three (especially if this principle is enshrined in the mission statement). Anyway, we already have upgrade scripts, they are just embedded in the existing scripts rather than a standalone program. The SharedCode will achieve the same goal (of always being able to use your historical data), only it will work for plugins as well.

I feel a little badly, it seems that lately I criticise every second assertion you put forward. I really don't disagree with you that much Peter. I largely agree with the central principles you use to guide the TWiki project along. Always being able to use older data is a good example. I just quibble with the idea that introducing upgrade scripts automaticaly breaks the principle. Particulars I argue about, principles, not so much.

• No problem at all, in fact constructive feedback is stimulating -- PeterThoeny - 07 May 2004

-- MattWilkie - 04 May 2004

May I ask how we know the forward compatibility features currently in the code still work? I'm not aware of any tests, but I can easily be corrected.

While I like the "seamless forward compatibility" concept, I do question how much of this legacy code TWiki has to carry. This question is especially relevant when the nature of the legacy code prevents us from moving forward. Having been a Clearcase maintainer for some years I am well aware of the problems with upgrade scripts, but in most cases it was well worth applying them simply to reap performance/functionality benefits. If the only issue is recovery of backups, then I would assert that as long as each TWiki release carries a full set of upgrade steps from any point in the past that can be easily applied by the restorer, and the route has been tested, I see no problem with that.

The worst possible scenario is when the compatibility scripts don't work, and that is as likely to happen with forward compatiblity in the core code as with an upgrade script (more likely, IMHO, as upgrade scripts are written only once, but the core code is touched constantly).

-- CrawfordCurrie - 07 May 2004