Stop Shipping TWikiUsers

TWiki currently (<= 4.1) ships with a populated Main web containing a draft Main.TWikiUsers and Main.TWikiPreferences. This is a bad idea.

Here's why. Consider the scenario for an upgrader. The simplest possible way for someone to upgrade their TWiki is to unpack a release package over the top of their existing install. If they do this currently the shipped TWikiPreferences and TWikiUsers will overwrite their locally customised versions. This is unexpected, bad behaviour.

Note that the same problem exists for customised default plugin topics, though I think we have almost eliminated the need to customise plugin topics now (at least I hope we have!)

I propose that Main.TWikiPreferences is removed from MANIFEST, and that the user management is recoded to survive with a missing Main.TWikiUsers, which would allow us to remove that as well.

You might be able to think of other site customisation files that require the same treatment.

-- Contributors: CrawfordCurrie - 11 Jan 2007

Discussion

I would LOVE this to happen, as it would remove one of the nasties that I would have to deal with after the TWikiOnDebian package goes into the upcoming release. Even better would be to make all distributed files readonly somehow.

-- SvenDowideit - 11 Jan 2007

Yes, we have discussed this several times and it is a dilemma.

• On the one hand, we want to ship a release that is stand alone and works in some rudimentary way with as little user work as possible.
• On the other hand, we do want to support the use model of upgrading by installing on top of an existing installation.

As some of the topics that should not be overwritten in an upgrade must be present in an installation (I think), they should be written in a fresh install.

In the past, we discussed the following strategy:

• Have more than one place where the TWikiUsers topic is stored (and other such topics). In TWiki.TWikiUsers we could ship the default version while the local site would update Main.TWikiUsers. The latter should take precedence, and so both scenarios above would work, provided we don't ship that version with an installation.

We have already implemented that strategy with respect to the preferences (almost, as we still shop a Main.TWikiPreferences which will overwrite an existing preference file--I think the reasoning here was to put the FINALPREFERENCES statement here). With the users file we need to do some more work to allow for such. I don't think we actually documented that the preferred way of customizing the preferences is in Main rather than TWiki.

Either way, I think what really is needed is that

• We identify all files that are user customized and needed in a fresh install
• We implement a scheme of not overwriting user versions and document it.

With respect to plugin topics, I believe it is not required (in principle) to modify them now. Of course, to really make this work the plugin writers need to start using config spec for some settings that should not be modified by the casual user.

-- ThomasWeigert - 11 Jan 2007

These are good ideas, but definitely too late for the 4.1.0 release.

One way to solve the problem is to create a global list to map new topic to template topics. For example, define this table in TWiki.NewTopicMapping:

| %MAINWEB%.TWikiUsers | %TWIKIWEB%.TWikiUsersTemplate |
| %LOCALSITEPREFS% | %TWIKIWEB%.LocalSitePrefsTemplate |
etc...

Then, when a user creates the Main.TWikiPreferences topic it is based on TWiki.LocalSitePrefsTemplate.

-- PeterThoeny - 11 Jan 2007

That is a cool idea.... is this feature implemented already?

-- ThomasWeigert - 12 Jan 2007

No, but is easy to implement. Lets follow-up in WebTopicEditTemplateMapping.

-- PeterThoeny - 12 Jan 2007

Rather than mapping templates (which only works for new topics, doesn't help existing installs, requires a lot of new code and - at least to my simple brain - is very obscure and complicated, why not just do what is described above (by Thomas?) and fall back to a topic in TWiki web if the topic in Main is missing? It sounds simple, elegant and consistent to me.....

-- CrawfordCurrie - 12 Jan 2007

This fall back works for TWikiUsers, but there might be a problem with TWikiPreferences: We might need some FINALPREFERENCES setting which needs to be in Main.TWikiPreferences rather than TWiki.TWikiPreferences and thus we would have to have a minimal Main.TWikiPreferences, if it is not present. We cannot get the FINALPREFERENCES in TWiki.TWikiPreferences, as that would mean those could not be set in Main.TWikiPreferences.

We should double check whether that situation still exists (i.e., are there preferences that must be finalized that are defined in TWiki.TWikiPreferences). Otherwise, the "fallback to TWiki web approach for certain topics" would be the most consistent route to go as we could handle all these topics in that manner.

If that could be done, it probably would be worth doing in this release, as it would ease the upgrade for everybody...

-- ThomasWeigert - 12 Jan 2007

Crawford, call it "very obscure and complicated" or whatever you like, I think you missed my point. We do not ship Main.TWikiUsers (e.g. no user content is overwritten), but we ship a new TWiki.TWikiUsersTemplate that is used when needed.

Actually, there is exactly one location where template topic is read, TWiki::UI::readTemplateTopic, so "requires a lot of new code", "number of places that would have to change in the code" does not reflect the reality.

-- PeterThoeny - 12 Jan 2007

There are actually a number of topics that should be omitted in the distribution and driven by a template:

• need default content based on template:
  • Main.TWikiUsers
  • Main.TWikiPreferences
  • Main.TWikiAdminGroup
  • Main.UserForm
  • Main.UserHomepageHeader
  • Main.UserViewTemplate
  • Main.WebHome - and all other webs in distribution
  • Main.WebLeftBar - and all other webs in distribution
  • Main.WebNotify - and all other webs in distribution
  • Main.WebPreferences - and all other webs in distribution
  • Main.WebStatistics - and all other webs in distribution
• better in Main web, with template in TWiki web:
  • TWiki.TWikiRegistration
  • TWiki.NewUserTemplate
  • TWiki.ChangePassword
  • TWiki.ResetPassword
  • TWiki.ChangeEmailAddress

There are probably other ones I missed.

Here is an idea: A number of topics have to be present in a TWiki installation, but must not be overwritten on an upgrade, such as Main.WebHome or Main.TWikiPreferences. We could add one more logic to the WebTopicEditTemplateMapping: Besides creating the topic on demand (when clicking on questionmark link), there should also be a "create if not exist" on first TWiki view. That is, running view the first time after configure, create all pages that are needed in a TWiki installation. This helps also on upgrade. Imagine a WebXXX that is new in a release. That topic will be replicated to all webs the first time you view a topic after upgrade.

-- PeterThoeny - 13 Jan 2007

Sounds like a convenient features, but I fear that this will cause a performance hit. We need to now on every view check whether the topics exist unless we can save the status in a setting so that this is reduced Boolean check. I worry about what happens if somebody creates a web with missing topics. If we want to automate checking there to, that would be a burden on the view routine.

If we can overcome the FINALPREFERENCES issue, I am still more in favor of the "fallback to TWiki" solution.

In either way, it would be great if we could have the Main web full with user modified stuff, while the TWiki web would not be user modified at all.

-- ThomasWeigert - 13 Jan 2007

The "fallback to TWiki" supports the TWikiPreferences case, but not other ones such as replicating a new WebXXX to all existing webs.

On performance: I was not specific enough. My idea was that the view script does the check only the first time after the configure script is run. Technically, the configure script could create a semaphore file, the view script takes action only if the semaphore exists.

Alternatively, the configure script could do that task. Although I feel more comfortable if there is no dependency on the configure script.

May be the view script can check for the $TWiki::VERSION and take only action if it changed? Yes, that looks more robust, and performs well.

-- PeterThoeny - 13 Jan 2007

Hmmmm..... I'm putting my "obscure and complex" face on again

I understand what you are trying to do, but is this really the way we want TWiki to behave? Sounds like a dose of creeping featuritis (yes I know the pot is calling the kettle black, sorry). On specifics;

1. TWikiPreferences already has a fallback, from Main to TWiki. All we need to do is remove Main.TWikiPreferences from the package.
2. TWikiUsers is not created from a template topic, it is written by the TWikiUserMapping code. That's what I meant by "a number of places" (there may be others). Other user mappers don't use it.
3. I thought we already had templates for the Web* topics (viz the topics in _default)?
4. TWikiAdminGroup is a good point, but I think I'd rather have that generated from configure if it doesn't already exist.
5. I'm not that familiar with the Main.User topics so I can't comment on those. But how many sites really customise these?
6. I'm definitely against anything that has more "side files" or "semaphores". The more state you add, the higher the risk of failure. If you insist on doing something like this, do it in configure.

Having been uniformly negative so far (sorry) I'm going to sound a couple of positive notes; first, I can really see the value of generating some of these topics from configure if they don't already exist in an installation (e.g. generate TWikiUsers if the TWikiUserMapping manager is selected, generate TWikiAdminGroup from a prompt for the name of the admin user(s)). Like it or not, succesful configuration depends on configure already. Second, I think the idea of templating new topics driven by the topic name is an excellent one; however I don't think the core should depend on it.

-- CrawfordCurrie - 13 Jan 2007

1. We can't simply remove Main.TWikiPreferences without a mechanism to have it created automatically. As Thomas pointed out earlier it contains some FINALPREFERENCES that cannot be set in TWiki.TWikiPreferences
2. If there is code that creates TWikiUsers by hardcoding it should be cleaned up. Much more transparent and easier to maintain if created based on a template.
3. We already have _default topics. The idea is to have an automatic upgrade of existing webs on TWiki upgrade
4. A hard dependency on configure is too limiting. For example, other distributions such as TWikiForWindowsPersonal could be shipped without a configure script (it runs after unpacking it)
5. There are many system topics (as I pointed out above) that currently need to be touched by the admin, making an upgrade harder
6. Good point on avoiding semaphores. Hence the argument for the view script taking action if the $TWiki::VERSION changed (or something else if positive minded community member has a constructive idea)

-- PeterThoeny - 13 Jan 2007

I think that the configure script has become essential and that there will be no twiki variants without it. Yet, if such is really required, that variant would have the user create many things by hand (the site local files, the configuration files in Main discussed here, etc.).

-- ThomasWeigert - 13 Jan 2007

From a practical point I think that we should not distribute configuration activity to too many places, so I like the idea of having the configure script perform the creation of these topics. It appears to be the simplest approach of those discussed also.

Thus I suggest that

• We concentrate all configuration topics that need to be modified by the user in Main
• We do not ship those topics with the release
• We do include templates for those topics in the release
• The configure script will create those topics from the templates if they are not already present in Main.

We can extend this to other webs so that configure will run through all webs and check for the presence of all necessary files and create them if needed.

-- ThomasWeigert - 13 Jan 2007

First - 4.1 is generated tomorrow. This is out of scope of 4.1.

Second. The above proposals suggests fixes for the right problem.

But the real problem is that it is such a pain to upgrade TWiki - PERIOD and the preference and user topics is just a small part of this.

What TWiki needs is something that enables an admin to...

• Unpack the distibution in a directory
• Run configure from browser
• Being guided through the steps of an installation by an installation wizard.
• Then optionally select the data and pub directories from an existing TWiki installation.
• The wizard now takes the admin though the steps of an upgrade where the admin can choose to perform the step or skip it.

We would need to work for a while with ideas for such a wizard. The Main and TWiki webs should give the admin the chance to select from a list topics that you may opt to either overwrite or not overwrite. Or maybe even a merge.

And for all the standard webs you should simply get the list and tick which ones you want to import and then the wizard copies the data and pub directories to the new installation.

Such a wizard approach would really make an impact both to the new installers and the upgraders.

And it is natural that such a wizard is initiated from configure which has become the one-stop place to shop for configuration. More and more plugins use configure for settings. Configure is the new easier way to install plugins. You cannot really talk about a TWiki without configure anymore.

I am thinking about making a special upgrade version of the 4.1 zip where all the "I do not want these files overwritten" files are either omitted or put in a special directory. I would hand craft such a zip.

I already have the list of the files that should be considered.

-- KennethLavrsen - 13 Jan 2007

Ken, agreed on the date, unfortunately... albeit I would suggest that we consider this feature as an early TWiki.4.1.1 as most users probably won't upgrade right away to TWiki.4.1 anyway...

But then, the "TWiki updgrade is such a pain" does not resonate with me (at least not for the reasons you cite)...

Here are the problems I perceive...

1. Configuring the web server properly
2. Figuring out the protection schemes and putting the right .htaccess files together
3. Getting TWikiUsers, TWikiAdminGroup, etc., moved over properly
4. Updating TWikiPreferences and all the web preferences properly to include local configuration
5. Copying existing webs and updating them with any new or changed files
6. Updating local template files
7. Applying local patches (and updating them to match the new release)
8. Deciding which plugins to install
9. Updating needed plugins to the newest release (both from TWiki.org and locally maintained plugins)

Many of these issues we won't be able to address. However,

• we can tackle (3) and (4) easily as discussed above.
• a wizzard can definitely help in (5), but it must both copy the web and update any changed files (web preferences, web statistics, etc., if applicable)
• the new template path will help in (6), as will the simplification of the default templates that Sven is working on.

As I said, I'd quickly do the first. The second is a nice feature for 4.2, and the latter hopefully will be resolved as well.

-- ThomasWeigert - 13 Jan 2007

For this release, what about having two zip files: one that has all the files minus the "I do not want these files overwritten" files, and one zip that has only the latter. An upgrader would use the first zip, and a new installation would then also use the second zip.

-- ThomasWeigert - 13 Jan 2007

For a newbie installer things should not be made more complicated than they already are. Splitting things in two is a step in the wrong direction for helping newbies getting started.

It is better to have a full release in ONE zip and make an upgrade zip where the "upgrade issue files" are by them selves.

The upgrader is familiar with his installation structure and can much better cope with this.

For the newbie there is already maybe 50 new things to understand and handle and I want to give them all the help they can get to get started properly

-- KennethLavrsen - 13 Jan 2007