Split The TWiki Web

this is one part of DefaultWebsInSubversion, a move to reorganise all of the TWiki and Main webs (as released)

After his/her new twiki site is installed and running, one of the first things the new administrator would like to do is separate the "day to day help docs" from "what's needed to keep things running" and "only for upgrades and extensive customisation" topics. This manually compiled list is a first step in that direction. The next step is to move these into two, or perhaps three, separate webs and see what breaks. The goal is to a) not overwhelm end users with reams of info they don't want and don't need, b) group often changed things together so they are both easier to manage and upgrade, c) identify those pages which are not really needed and can be safely removed, if desired.

• TWikiDocumentation - topics in the TWiki06x01 web which is could be considered "just" end-user documentation, WelcomeGuest and TextFormattingRules for example.
• TWikiConfiguration - topics in the TWiki06x01 web which are used for configuration and system setup. TWikiPreferences and TWikiPlugins for example. Also includes reference material likely only of interest to the site administrator.
• TWikiUsers - contains the registered Users

There are some pages which don't cleanly belong to one category or another. Particularily plugin pages, are they more reference documentation, or configuration? At this point I'm more interested in splitting into End-User and System webs than anything else.

The pages in the list are based on the 2004-09-02 release, which is why there are some broken links (and many missing pages).

-- MattWilkie - 06 Feb 2005

Recognizing that there are long established twiki deployments who will not feel the need to split in this way, I request that a TopicClassification be added to pages in the TWiki web with values like "!EndUser", "System" and "Reference". Also now that I think about it "Discussion" and "!NotOfficial" for all those topics that are now tagged 'This topic is not part of the release documentation'.

I'm willing to do the heavy lifting required to classify all the existing topics.

-- MattWilkie - 06 Feb 2005

just to make sure - my intention in DefaultWebsInSubversion was to work towards the splitting and moving being scripted into UpgradeTWiki - it would be optional for the user, but would give us a simple, repeatable (and reversible?) move.

-- SvenDowideit - 07 Feb 2005

Why not rename this topic to something like ReorganiseWebs and refactor the relevant content in DefaultWebsInSubversion into this topic?

-- SamHasler - 07 Feb 2005

The auto-generated source code documentation (generated by gendocs.pl) goes into the TWiki web at the moment, adding to the confusion.

-- CrawfordCurrie - 19 Feb 2005

Sam I branched the discussion in order to remain tightly focussed on this one part of it. If this turns into a general how to re-org all the webs nothing at all will be agreed on or implemented (an assumption based on past experience. I'm willing, hoping even, to be proven wrong).

-- MattWilkie - 20 Feb 2005

The issue of "day-to-day" docs vs "what's needed to keep things running" and "only for upgrades and extensive customisation" topics isn't a brain buster, is it? We have the ability to create a classification sceme for the "TWikidocs" as Matt says.

Making the documentation easier for both end users (new and not so new) and for the administrators is surely a good thing. Having a web that makes is clear "Documentation lives here" is one of those simple usability things.

Really, what does it take to make this happen?

-- AntonAylward - 23 Feb 2005

very little, i found I started to hack up an addition to upgradeTWiki that ends up calling

createWeb(TWikiConfig)

moveTopic("Main.ManagingWebs", "TWikiConfig.ManagingWebs")

-- SvenDowideit - 24 Feb 2005

Thank you Sven but thay's not what I meant - I meant at the "policitcal" level not the coding level. Add in Matt's TopicClassification ...

And anyway, there's still the effort of going through the TWiki web identifying the topics for your "one line fix-up".

All that warants more than a smiley.

-- AntonAylward - 24 Feb 2005

ah, I was thinking that initially, I'd just add an optional bit to the upgradeScript, and people could choose to use the new layout, or keep the old. This way it would be possible to continue using and upgrading an old setup, or to upgrade to the new layout. (though you're right, something would need to be decided for the new installations - but we are working towards having an install script / interface) (Again - this is all in DEVELOP, i don't have the time to get involved in MAIN at the moment)

-- SvenDowideit - 24 Feb 2005

I have commited an initial version of this to DEVELOP, and in the process have begun to move around the cairo and dakar upgrade code. (which now simply does not work - its a work in progress)

-- SvenDowideit - 27 Feb 2005

I would like to take a step back and understand what we would like to accomplish with this proposed change. The key question is to make an installation and upgrade easy. At first it looks like a good idea to split up the content into a doc and config section. What do we gain? Simple way of upgrade based on TWiki web.

On the other side, a real upgrade needs to update the config topics transparently, ideally use the new page as a template and copy over the old config values, then create a new revision in the existing config topic (so that the admin can see what changed)

One drawback of splitting up the TWiki web into two webs is that we get yet more webs. I have seen a consolidation of webs, e.g. a trend to reduce the number of webs (for content). This is for easy of maintenance, cross-linking and search. I have seen also installations that collaps the Main web and TWiki web into one web.

Plugin topics serve two purposes, doc and config. Where would they go if we split the web? In any case, if we rename the TWiki web we would need to ask 140 Plugin authors to repackage their Plugins...

With an intelligent config update mechanism as described above we would not need to split the web. Opinions?

-- PeterThoeny - 01 Mar 2005

We have an intelligent config update mechanism, UpgradeTWiki. However because local mods can occur, that means there are almost always a few rejections of the upgrade patches. Because read-only docs and read-write config is all mixed up together, it is quite scary working out what needs to be looked at carefully ("nursed") versus what can simply be copied in from the release package ("stomped").

I'd love to separate the read-only docs from the read-write config, if only to divide-and-conquer this problem.

BTW why "TWikiDocs" and "TWikiConfig"? What is wrong with "Documentation" and "Configuration"? Sounds a lot less nerdy.

Oh, and plugins topics should be documentation only, IMHO. Personally I think all plugins configuration should be in one place, not scattered through dozens of different topics.

Oh, and note that there is some risk that a plugin has explicitly used the web name "TWiki" somewhere. I'm pretty sure at least one has used the name "Main".

-- CrawfordCurrie - 01 Mar 2005

> BTW why "TWikiDocs" and "TWikiConfig"? What is wrong with "Documentation" and "Configuration"? Sounds a lot less nerdy.

For the same reason we got rid of the web that pre-dated Sandbox (was it called Example, I don't recall): to ensure they do not clash with anything the administrator might want to set up.

The TWiki namespace is pretty much the only thing we can guarantee the user won't mind us using: if they want to rename TWikiConfig they can do so so long as we use e.g. %TWIKICONFWEB% consistently.

TWiki has assumed that claiming "Web*" for its own purposes is also okay: its not - there is a software company called WebMethods, for example.

I do agree we need to split the TWiki web.

-- MartinCleaver - 01 Mar 2005

PeterThoeny said "Plugin topics serve two purposes, doc and config. Where would they go if we split the web? "

I was in my lsat year of an Enginnering degre when I first saw the list of "Fifty reasons why not". I've seen it reprinted in many "Book of quotes and rules" since then.

The "We've never done it that way before" is always number one on this list. See, as an example, http://www.flex.net/~mcgovern/fifty.html

I'm no suporter of "Change for the sake of Change". As I grow older I'm becvomeing more conservative, but I can't agree with Peter on this. The argument about isolating what is site/system dependant and what is not is simply too strong. We've already started that with "LoclaSite.cfg".

Personally, I'm sick of having to run "sdiff" and the like on the various Preferences as each mod comes out. And as to the matter of plugins, perhaps the config and the documentation should be separate. See also http://www.vista-development.com/rails.htm for an extreme case of "We've always done it that way" where the space age is limited by the old Roman (> 500 years BC) way of doing things.

Form, one should recall, refelcts function. The funtioning of Twiki has evolved; its form should evolve to match.

-- AntonAylward - 01 Mar 2005

Mixing docs and configs in the TWiki web is a big hurdle for upgrading the system. So separating them is a good idea. As to the location: I favor the Main web. By definition, it already contains site specific configurations for users, groups and permissions. Other bits fit naturally.

A configurable sequence of reading the preferences might ease migrations. Currently, it is hardwired to:
TWiki -> Web -> User.
The above proposal would mean:
Main -> Web -> User.
If we want TWiki documentation to provide product defaults (and accept the performance hit) the sequence would be:
TWiki -> Main -> Web -> User. If a modified TWiki has to be upgraded, the installer just moves colliding pages to Main, then copies the new product files to TWiki.
And yes, plugins should not be configured in their documentation page, but non-defaults should come from WebPreferences etc.

-- PeterKlausner - 01 Mar 2005

Peter mentioned "consilidation of the webs".

Indeed, some people have consolidted it all down to a single web!

But lets think about this from the POV of ENTROPY. Why Entropy? Becuase its a measure of information content and of the balance between confusion and coherency.

Its easy enough to do a mv Main/* Sandbox/* TWiki but its not easy to do the converse, to seperate them out again. OK, so its not as bad as trying to get the sugar out of your coffee.

That's Entropy. Its an increase in the 'confusion' of the system.

Our long term efforts have bee to fight an increase in entropy. If you look at the Left Menu bar for TWiki:Codev you'll see a result of that effort, a classification of topics. Its quite agressive. Even in the TWiki web we can already see that there is a classification into the operational documents ("Maintenance") and the informative ones ("Reference"). The classification in the Main web is much simpler.

And lets face it, the Main web is not the main web, is it? I've seen mention of people setting up many dozens/hundreds of webs. But the Main web stays there as the Users & Group information. As such its a perfect case of form reflecting function. All its needs is to be renamed "User".

Why? Because "Main" is not a name that reflects its function. (Isn't it good practice to name variables and objects and functions in a way to reflct what they 'mean'?) And lets face it, webs with names involving "Documentation" and "Configuration" or recognized abbreviations thereof would also fr that mold.

So what do we have now?

• Where does the system documentation live?
  In the TWiki web
• Where does the system configuration files live?
  In the TWiki web
• Where does the Tip of the Day live?
  In the TWiki web
• Where does the per-plugin documentation live?
  In the TWiki web

And so it goes ... Its like a religious litany ... http://www.columbia.edu/cu/augustine/arch/bvm/loretto.html

-- AntonAylward - 02 Mar 2005

UpgradeTWiki --moveTopics in DEVELOP now makes the TWiki and Main webs empty, ready for removal by the user (unless they have created topics in them. The distributed topics are moved into TWikiDocumentation, TWikiConfiguration and TWikiUsers. The choice of web names is deliberate, in that it frees up the entire Web namespace to the user. I hope at some stage for us to be able to prompt the user to create a web for their use when they first install.

the idea is simply to remove the cruft that an experienced TWikiAdmin wades though to find a configuration setting, and to reduce the number of topics a new user needs to search through to find the information they need.

-- SvenDowideit - 12 Mar 2005

one more reason for separating the conf files from the rest: when you're running more than one TWiki version on the same data, you probably may want to share the same configuration, whereas each version has its own docs. all right, maybe You don't want to share the configuration, I do.

is there any problem with putting also Plugin configuration data in the new TWikiConf web?

-- MarioFrasca - 12 Mar 2005

no, i don't see any 'real' problem with putting the plugin config data into the TWikiConfiguration web - though this would be simplified if there was a PluginInstaller, as the current installation method allows the plugin creator to put documents anywhere they like.

-- SvenDowideit - 12 Mar 2005

Sven, I'm running a semi-production Dakar-alaph 380-something. I like the idea of the split. But can I run UpgrdeTWiki on it?

Maybe a sub-project for any one interested is to go though the plugins and split the pages currently in the TWiki web into their documentation and configuration compnents then upload them.

-- AntonAylward - 13 Mar 2005

yes, you can run

./UpgradeTopics --moveTopics bin/setlib.cfg

just be warned, it also shows up some bugs in the renameTopics code, so you loose some links (the [[ ones, and a few things like the css links for Pattern skin in TWikiPreferences will need hand editing)

-- SvenDowideit - 13 Mar 2005

The problem with using "TWikiConfiguration" and "TWikiDocumentation" and "TWikiUsers" and of using the conversion script is two-fold.

1. We have %TWIKIWEB% and %MAINWEB%, but when the move is done by the script I find that ther are hard-coded referedcnes to "TWikiUsers" and "TWikiConfiguration".
2. Thw "Welcome" topic has test that begins "You are curently in the %WB% web". When %WEB% is expanded for "TWikiUsers" and "TWikiConfiguration" they look like WikiWords but are undefined so have the little questionmark. Ugly

In addition, there is the matter of the configuration web. It too gets hard coded becuase there is no %DOCWEB% defined in the lib/TWiki.cfg

-- AntonAylward - 19 Mar 2005

Dropping this from DakarRelease due to lack of progress/interest. If you want it to live again, you are going to have to do some work on it.

-- CrawfordCurrie - 30 Apr 2005

I am not in favor of creating more system webs. Kiss: Keep the number of webs as low as possible and add structure within a web (with TWikiForms etc)

-- PeterThoeny - 01 May 2005

Why are big webs mingling different stuff simpler? You need to add structure to depict the information you actually need. Kiss: Keep TWiki's documentation separated from system management

-- MichaelDaum - 01 May 2005

Peter's argument about minimizing webs can easily be refuted by following it to its logical conclusion - doing away with webs altogther and using solely a category management system.

From the point of view of "Computer Scientists" there's no difference what so ever.

But from the point of view of ordinary users, paritioning in segregation is perfectly natural.

I have a draw for socks, one for underwear, another one for shirts. I do not have one big drawer into which everyting is thrown and I have to rummage around in looking at labels!

It is stucture that leasds to simplicity. The simplicity is in the use - the verb, the dynamic, not the noun, the static.

CC: there is no lack of interest. My users are complaining when they go to look for documentation:

• Why is documentation in the "TWiki" web?
  • Why isn't there a documentation or help web?
  • Why is there all that other junk in with the documentation?

Form needs to refelct function.
There are two distinct functions here, that of searching documentation and that of administering the system. (In reality there are actually two types of documentation - system and application, but that's beside the point.)

End users don't need to know and shouldn't see the mechanisms of the system. End users are application users - stop thinking of them as the same kind of people who are active here.

Peter - please note. I'm not advocation splitting the TWiki web at TWiki.org. Here, we are developers. The situation is different.

SvenDowideit's code works - though it needs some assistance and some topics need cleanup. As CC says, not everyone has been a good topic writer.

I've run the code to produce the three webs on a Dakar system on my internal net. I'd like to run it on my production system, but that would mean that every time I run a svn up I would have to split it out all over again.

Form refelcts function.
The usage demands and patterns and hence the access controls and visibility are quite different for configuration control/system management and documentation on the other. In the extreme case the application side simply cannot see, never mind access, the system management side.

The aberation of the fully open system caaame with single user PCs. Even Microsoft is now beginign to realise that importance of seprating admin functions from end-user functions. All my corporate clients have their PC workstations configured so that the end user - the applicaiton user - had no admin access on their own workstation and cannot even see the system management directories and functions on the servers.

PeterTheony suggests using TWikiForms to categorise. Sorry Peter, I have better things to do. If I keep to your KISS principle and minimise the number of webs, I have to use forms for categorization and can't use them for applicaiions. Even here in the Codev web every form has to have a TopicClassification field. Its weak.

As I said, the logical conclusion of Peter's argument is to merge all the webs here at TWiki.org. as well. It hasn't happened and I'm sure it isn't going to happen.

In reality, its more like an elastic band; even with the WebForms and TopicClassification it can only stretch so far.

Form refelcts function;

-- AntonAylward - 20 Jun 2005

wouldn't SubWebs (whether in the form of LogicallyNestedWebs, HierarchicallyNestedTwikiWebs, or MultiLevelWikiWebs) be a solution to what is essentially a NamespaceControl issue? MegaTWiki had this feature long ago, as did TWiki proper (though it was an unofficial feature, and eventually code changes broke it). some implementation ideas are located at DataStorageForMultiLevelWikiWebs and it should also fall out of a proper implementation of TopicObjectModel.

-- WillNorris - 21 Jun 2005

on TWikiIRC, CrawfordCurrie correctly pointed out to me that WikiBadges are another NamespaceControl mechanism and that a possible solution also lies with WikiBadges

-- WillNorris - 21 Jun 2005

personally I prefer to use a seperate web not for NamespaceControl, but rather to seperate and simplify TWikiApplications

for example, at HsaSystems we had 4 webs (Hsa, Tasks, Support, Qms), where only the Hsa web was a normal discussion web like we have here. the other 3 were specific application webs for managing our Tasks and Support Data (basically a TWikiForm Database with not freeform text) and the ISO9001 procedures web, which was a mix of freeform text for the procedures, and a TWikiForm with plugin to enable processes to be managed and authorised by QA only, while allowing everyone to update the Draft version (essentially there was an authorised revision that was what you viewed by default, while editing happened on the head revsion)

this is similar to seperating out the Docco, Config and User topics into seperate webs.

surely it would not be difficult to implement a virtual web system, which uses WikiBadges / TopicClassification as the "web" name

eg you could find this topic in http://twiki.org/cgi-bin/view/FeatureRequest/WebIndex even though there is no FeatureRequest web....

-- SvenDowideit - 21 Jun 2005

Another one to resolve.

-- MartinCleaver - 11 May 2006

Let's step back a little. The point of this is to make upgrading easier, by having... "official" pages and "user" pages, so that the "official" pages can be replaced on upgrading, but the "user" pages can remain untouched. Perhaps one could take a leaf out of PmWiki's book: Like TWiki, PmWiki has namespaces (TWiki calls them webs, PmWiki calls them groups) where each namespace has its own directory (and pages go into the directory). What this discussion seems to be focused on is "Okay, what are we going to call the 'official' namespaces?"

But PmWiki has a different way of separating the "official" from the "user" (and this could be helpful for looking at the SubversionAsStore idea too). PmWiki uses an array of PageStore objects, and looks through the array, asking each PageStore whether it has the given page, and the first one found is the one that's used. In the default PmWiki configuration, there are two PageStore objects -- one maps onto the wiki.d directory, and the other maps onto the wikilib.d directory. The wiki.d directory contains the user pages, and the wikilib.d directory contains the official pages. And the clincher is that the wikilib.d directory is "read only" -- if a user edits a page which originally came from wikilib.d, the new version of the page will get saved in wiki.d, leaving the "official" page safe and untouched. Since the wiki.d directory is looked at first, the "user" version of the page is the one that's used (thus, preserving any customizations that were done). But if the page doesn't exist under wiki.d, it then looks in wikilib.d. This makes upgrading much easier, since there's no moving of pages, no trying to figure out what pages are readonly, what webs are readonly etc. All you do is (a) when creating a release, move the current "official" pages into the "official-pages-dir" (b) when upgrading, don't touch the user-pages-dir at all, just copy all the new "official" pages into your official-pages-dir.

This could make things easier for having different backend storage solutions too, because one could actually have a mixture of them; for example, you go through your PageStore array, and one could be RCS directories, and one could be a MySQL database, and it would use whichever "page" it found first.

-- KathrynAndersen - 19 Mar 2007

yeah, It would also make the _default topics simpler - instead of copying the WebSearch topic to create a web, it gets 'mixed in' from the _default web - or more likely, would allow the removal of the _default web altogether, replacing it with a 'set of topics' from the TWiki web.

-- SvenDowideit - 19 Mar 2007

The FindElsewherePlugin already supports a search path for topics.

The TWiki web is now more or less a read-only web, configuration of the site and of installed plugins can now be done in Main.TWikiPreferences.

-- PeterThoeny - 19 Mar 2007