TWiki configuration is a tremendously complex topic; unneccessarily complex, probably, but a problem no-one has ever solved. There are three aspects to TWiki configuration:

1. Configuring the webserver to serve a TWiki (apache configuration)
2. Configuring the server (file permissions, backup regimens etc)
3. Configuring TWiki (the LocalSite.cfg file and TWikiPreferences topics)

This topic concerns itself with (3). (1) and (2) are discussed elsewhere.

How TWiki bootstraps itself: the @INC path

Any CGI program that is more than a simple script needs a way to find its libraries and configuration data. Both the TWiki libraries and the TWiki configuration files must be found on the perl @INC path for TWiki to work. (Actually this is not exactly true. The TWiki configuration file could be processed with an absolute file name: do '/absolute/path/to/config'; (require would do as well, albeit this isn't well documented). This gets relevance if you want to consider "multiple TWikis from the same codebase")

TWiki sets up the @INC path using setlib.cfg, a perl script located in the CGI bin directory. This script is included by all TWiki CGI scripts. setlib.cfg supports the definition of two things; first, a single path to the TWiki library directory (the configuration files are tacitly assumed to be stored here as well) and a multi-directory path to extra modules that TWiki (or plugins) may require. This is usually used to point to CPAN libraries.

The @INC path must not be confused with the environment variable $ENV{PATH}. They are totally different. @INC relates to perl only. The environment variable related to @INC is $ENV{PERL5LIB}. Unfortunately it is useless for TWiki because $ENV{PERL5LIB} is ignored for programs running with taint checking (the -T switch).

How TWiki::cfg works

Cairo

Cairo uses perl do to include TWiki.cfg. TWiki.cfg is a perl source file that defines a bunch of global variables in the TWiki namespace. There is no automation. Documentation is in the form of comments in TWiki.cfg.

Dakar

Dakar continues to use TWiki.cfg, but it treats it as read only (in fact, it uses it as a source of configration variable declarations (type definitions and defaults), and nothing else). TWiki uses perl do on TWiki.cfg and a second, site-specific, LocalSite.cfg. All configuration information is encapsulated in a single global hash, called %TWiki::cfg.

The configure interface declarations and values from TWiki.cfg and LocalSite.cfg and presents them for editing in the browser. Plugins use the methods in BuildContrib to add configuration variables and type information to LocalSite.cfg, but plugin installation remains an essentially manual process.

Recent work

Recent work for 4.1 by CrawfordCurrie has refactored the configure interface in a whole bunch of pluggable modules. This means that a plugin or contrib can now install its own checkers, types, and even complete UI interfaces, and they will be picked up and automatically integrated by configure. LocalSite.cfg is now written under program control, so may no longer contain arbitrary perl. Also, because plugins can now be installed from the browser this has necessitated some changes to the way plugins configurations are handled. Specifically, support for plugins to add their own variable to the configure interface has been strengthened.

For those who know Dakar configure, here are the main changes. The goal was to reconfigure configure to allow plugins and contribs to provide their own configuration requirements to the interface.

1. All checkvar functions moved to modules in lib/TWiki/Configure/Checkers
2. Check functions for specific operating systems and webserver types also moved to modules in lib/TWiki/Configure/Checkers
3. Type handlers (prompting for values in configure, mainly, but also writing types) moved to lib/TWiki/Configure/Types
4. A new class of UI handlers introduced. Languages and Plugins made into UI handlers. General procedure for adding arbitrary UIs defined.
5. Configuration files loaded and written by a separate module, lib/TWiki/Configure/TWikiCfg.pm
   • Separation into declaration files (.spec) and *value files (*.cfg).
6. New methods in the installer script support hands-off installation

Taking a step back, here's the current situation from 1000ft:

1. LocalSite.cfg just stores values. Any declarations in this file are just ignored when loading into configure.
2. TWikiCfg.spec must be found on the @INC path, and must contain the declarations formerly in TWiki.cfg (the values are used as defaults in configure)
3. All other uniquely named *.spec files found on the =@=INC path will be read (i.e. once Fred.spec has been read once it won't be read again, even it occurs on the path again)
4. LocalSite.cfg contains all configuration values. TWiki only reads LocalSite.cfg (not TWiki.cfg). Configure reads both, for simplicity when upgrading.
5. Plugins and other extensions can add their own configuration requirements by installing .spec files on the path

I did consider keeping LocalSite.cfg as the values, and still loading TWiki.cfg, but it just seemed unneccesarily complex. As soon as I started trying to write user docs to explain it, I realised the approach just had to go.

How TWikiPreferences work

TWiki preferences are TWiki variables defined in TWiki.TWikiPreferences and Main.TWikiPreferences. Definitions in TWiki can be overridden in Main. The recommendation is to leave TWiki.TWikiPreferences as read-only, and only change TWikiPreferences.

In general, preferences defined in TWikiPreferences topics fall into three categories:

1. Those that should be modified by the admin only
2. Those that provide defaults that other users may override at web or topic level for local customisations
3. Those that provide constants useful throughout a site e.g. shortcuts to icons, colours etc.

In general, preferences in category 1 should not be preferences. Many have migrated to TWiki::cfg already, but this is an ongoing process.

What should be configurable?

TWiki is tremendously configurable, and it is this very configurability that makes it hard to install. Most of the time, configurability comes about because a developer lacks the knowledge, courage or conviction to define a constant, so they throw a new configuration item into the pool. Most of the time these "options" never waver from their defaults and indeed, making new configuration options creates new test problems, as every new configuration option exponentially increases the number of possible TWiki configurations.

Some initial work has resulted in options being tagged as "EXPERT" in configure. The next step is to pull those options out of the interface altogether, or at least only show them when "expert mode" is enabled.

-- Contributors: CrawfordCurrie, HaraldJoerg

Proposals

Split configuration information into declarations and values

Split the TWiki.cfg/LocalSite.cfg configuration information into type declarations and values. So

• TWiki.cfg would continue to hold the same content as it does today, but it would become declarations only.
• It would not be read by TWiki.pm.
• Instead, TWiki.cfg (and any .cfg files provided by extensions) would be parsed by configure, which would then write all the final values to a file (e.g. by using Data::Dumper to write the entire %TWiki::cfg hash to a location pointed to by an environment variable set in the apache config).
• LocalSite.cfg would be retired, lib would cease to contain any site-specific configuration files.

The obvious advantages of this are that it is simpler to understand, simpler to code, easily extensible, and will work much better for sites that want to use single-code-multiple-data.

Retiring LocalSite.cfg would mean that configure can easily auto-upgrade sites using it.

The format of how TWiki stores its syntax rules for configuration items (today it's a special sort of Perl comments) is created by developers and read by bin/configure, but should be totally irrelevant to the admin or user.

-- RafaelAlvarez, HaraldJoerg

I would vote against your proposal to dump the configuration procedure to a file pointed to by an environment variable. Environment variables create another dependency between web server and TWiki config, they are "tainted" (for a good reason), and it is another nasty source of errors when running unit tests or other command line invocations. Write it to a fixed location within the installation tree instead. If an indirection is needed, just have bin/setlib.cfg consist of one single line:

do '/absolute/path/to/config'

...and let bin/configure take care for the rest. One could get rid of the assumption that TWiki starts in the bin directory, which would be a blessing for unit tests, cron tools, ....

-- HaraldJoerg

A fixed location in the installation tree is what we currently have, and it is unacceptable because it "freezes" the code to a single TWiki. If you want to run multiple TWikis off the same codebase (different pub and data, but otherwise identical) a fixed location blocks this.

-- CrawfordCurrie

Good point. I've missed that requirement. But I think that "unacceptable" is a hard word, because when talking about enabling multiple TWikis off the same codebase we are talking about an enhancement. And a tricky one, too:

• "Same codebase" needs a web server configuration where e.g. view URLs from different TWikis point to the same file.
• Multiple TWikis need different data and pub dirs, so they need different values for those in their %TWiki::cfg hash.
• However, in a mod_perl environment, there is only one %TWiki::cfg, and configuration files are read only once.
• A mapping would be needed to find the correct data directories for a particular request, so you would either have to re-create the config hash per request (i.e. re-read the configuration file) or to create to an Über-hash, with =%TWiki::cfg=-like substructures for each wiki.
• If you have separate configuration files for separate TWikis, then finding the configuration files themselves is as difficult as finding the data directories. You can't put them into the data directories since they are assumed to contain the location of the data directories.
• If you have one Über-configuration file, then bin/configure is getting really complex because you need to do file locking in case of different twiki admins wanting to change their configuration at the same time. Such collisions are getting more likely if TWiki preferences or plugin preferences are replaced by configuration items.

Intelligent configuration can take care for that: While running bin/configure, the performance penalty of "discovering necessary fallbacks" doesn't matter. But if you want to pursue this enhancement in 4.1, it would in my opinion deserve a Codev topic of its own because it is likely to change TWiki behaviour at some edges.

-- HaraldJoerg - 05 Sep 2006

Move site level TWikiPreferences to the TWiki::cfg hash

1. Almost the entire content of TWikiPreferences should be moved to the TWiki::cfg hash. This is for three reasons:
   1. Reading and managing preferences in TWiki is horrendously inefficient
   2. The security of the configure interface is much higher
   3. Unless you are careful about finalising prefs, they can be overwritten on a web or topic level. This is dangerous and a source of FUD. However, I also think the current mechanisms, of having a set of default preferences and then a level of site preferences is a good one. I just think there are a lot of preferences defined in them that don't belong there.

-- CrawfordCurrie

I second the idea of include TWikiPreferences in future concepts, as well as a migration of preferences from plugin topic preferences to configuration settings, though I would expect the latter to be a "slow" migration process because of plugins' backward compatibility.

-- HaraldJoerg

Include bin/setlib and bin/LocalLib.cfg into refactoring

I'd like to consider bin/setlib, or better bin/LocalLib.cfg, together with lib/TWiki.cfg and lib/LocalSite.cfg:

1. Like the other configuration files, it is Perl syntax, directly executed by TWiki.
2. It contains data (directory paths) which are written only during initial installation, quite similar to some of the data which may be present in LocalSite.cfg, and which are the reason for the funny roundtrip of 1. LocalSite.cfg, 2. TWiki.cfg, 3. LocalSite.cfg (again).
3. The separation is a result of a developer's mind, not an admin's: The developer knows that the library paths are needed to bootstrap TWiki (including bin/configure) itself, whereas data directories can be set later by bin/configure. The administrator, however, has to create both the library and data paths on the command line (e.g. by unzipping a distribution), so there is little point in having him to declare one of the paths in bin/LocalLib and the other via bin/configure.

I think we agree that, during "normal" operation, TWiki should continue to be able to read all its configuration settings (including plugin settings in some future) with a simple Perl's do (or require) statement, possibly in a dedicated routine similar to readConfig introduced in r11384. This results in requirements for the file written by a bin/configure web interface:

1. It needs to be written where TWiki's user interface scripts in bin can find it. The bin directory itself (as in the case of bin/setlib.cfg) or the lib directory (where TWiki will find its libraries) come into mind. If it is in the bin directory, it could be directly executed as an URL (unless prohibited by web server configuration), however this would be no security hazard at all.
2. It needs to be writeable by the user id running the web server.

Hmmmm. Firstly, LocalLib.cfg is used to bootstrap TWiki, by setting the library paths. That is its sole purpose. No other configuration item affects the @INC path, and IMHO no other configuration item should. So LocalLib.cfg is probably misnamed, but it is required. Using the perl path to find TWiki configuration would be cleaner, but is not portable to all webservers (ISS doesn't support it properly). So I think we are stuck with bootstrapping. Of course we don't need two files, but that's the way it has worked out, unfortunately. What do you think the advantages of using a URL are? To me they aren't clear. -- CrawfordCurrie

• Sorry, I've never intended to mean "URL" (and changed the header accordingly). And your reminder of the multiple TWiki feature is indeed a warning sign against any attempts to join bin/LocalLib.cfg and lib/LocalSite.cfg. -- HaraldJoerg

Unclassifiable suggestions

• Move all file system related information (lib paths, data and pub dirs, ...) to one location. -- HaraldJoerg
• bin/configure is on a good way, regarding "intelligence" when discovering the sanity of an installation. If it is able to write all configuration settings to one (i.e. non-incremental) file, it could indicate that with a flag so that readConfig would not do the roundtrip. The configuration procedure could even take care for a "migration" by interpreting LocalSite.cfg, moving "actual" configuration stuff to the "true" config routine, and leave any "dirty tricks" section intact. -- HaraldJoerg

---

In order to support plugin *.spec files to be read, added code that goes through the plugin directories to look for Plugin.spec files (the current code looks only on @INC and plugins are not on that path; not sure what is expected to be found on @INC).

Currently this allows the plugin writer the freedom to pick the section where the configuration information shows up; probably better would be to enhance TWiki::Configure::TWikiCfg::_parse to take a section as parameter.

I have this also written for the TWiki 4.0 configure script, it would be good to merge that in as soon as possible, so that plugin writers can start migrating their configurations.

-- ThomasWeigert - 25 Sep 2006

I just wrote a Config.spec file for the LdapContrib starting from zero knowledge, sparce docu. Wow, this was so easy. This framework rocks. Here are a few things I came across:

1. how do I create a link from the configure page to the TWiki installation, i.e. to the docu of a contrib in the TWiki web. I tried <a href="$TWiki::cfg{ScriptUrlPath}/view...">read more</a> like this but this didn't work out though it works in TWiki.spec.
2. Although I get a red delta on some entries clicking on "next" says that there are 0 (zero) items to configure. Saving them will nevertheless alter the LocalSite.cfg
3. Even though I saved and go back to the configure page, there are still red deltas left. I was expecting them to go away when the data and the view are in sync, i.e. right after saving
4. I still tend to edit the LocalSite.cfg file by hand commenting out some options. For example I have the following for testing utf8 vs iso

         $TWiki::cfg{Site}{CharSet} = 'iso-8859-1';
         #$TWiki::cfg{Site}{CharSet} = 'utf8';
        

   When I save, this becomes

         $TWiki::cfg{Site}{CharSet} = 'iso-8859-1';
         #$TWiki::cfg{Site}{CharSet} = 'iso-8859-1';
        

   I would have expected "configure" not to touch the second line as it is commented out.

-- MichaelDaum - 11 Dec 2006