TWiki 'Dakar' Release

Key new features:

• Much simpler install and configuration
• Integrated session support
• Security sandbox blocks all possible routes for remote command excution on the server
• New editing model allows freer collaboration, without fear of overwriting other people's changes
• Multilingual UI
• Email confirmations for registration
• Hierarchical sub-webs

Most of the redesign, refactoring and documentation work in Dakar release was done by Crawford Currie^? (http://c-dot.co.uk). Other major code contributions were made by:

• Antonio Terceiro^? - internationalization framework
• Arthur Clemens^? - templates
• Florian Weimer^? - security sandbox
• Greg Abbas^? - integration of SessionPlugin
• Martin Cleaver^? - registration process, and automatic attachments
• Peter Nixon^? - hierarchical subwebs support
• Thomas Weigert^? - preferences editing

• Anton Aylward^?
• Lynnwood Brown^?
• Michael Daum^?
• Michael Sparks
• Sven Dowideit^?
• Will Norris^?

Note: In what follows, {This} (words in curly braces) refers to settings in the new 'configure' interface.

Notes for end users

Editing at the same time as other people

Dakar release introduces a brand-new strategy for handling simultaneous changes to a topic by several people. Instead of one person locking the topic, and other having to wait until they are finished, Dakar allows multiple simultaneous edits of the same topic, and then merges the different changes.

You probably won't even notice this happening unless you are changing existing text in the file at the same time as someone else. In this case, you may see TWiki inserting "change marks" into the text to highlight conflicts between your edits and another persons. These change marks are only used if you edit the same part of a topic as someone else, and they indicate what the text used to look like, what the other person's edits were, and what your edits were. For example, let's say you have a topic that contains this text:

Casablanca is
Humphrey Bogart's finest film.
Of all the gin joints in all the world,
you had to walk into mine.

The Maltese Falcon is
Humphrey Bogart's finest film.
Of all the gin joints in all the world,
you had to walk into mine.

To Have or Have Not is
Humphrey Bogart's finest film.
You know how to whistle, don't you Steve?
You just put your lips together and blow.

<div class="twikiConflict"><b>CONFLICT</b> original 5:</div>
Casablanca is
<div class="twikiConflict"><b>CONFLICT</b> version 6:</div>
The Maltese Falcon is
<div class="twikiConflict"><b>CONFLICT</b> version 7:</div>
To Have or Have Not is
<div class="twikiConflict"><b>CONFLICT</b> end</div>
Humphrey Bogart's finest film.
You know how to whistle, don't you Steve?
You just put your lips together and blow.

Merging only applies to text fields. When there are conflicts in field data in forms such as checkboxes, radio buttons and selects, the most recent change (usually your change) always wins, even if someone else has changed the form since you started editing. All changes are of course available from the topic history.

Because there are cases where you actually want to avoid overlapping edits altogether (e.g. if you are changing forms data) TWiki will warn if you attempt to edit a topic that someone else is editing. It will also warn if a merge was required during a save.

Internationalisation

TWiki will now pick up the language you are using in you browser, and try to present system messages in you chosen language, if it is available. If your preferred language is not available, TWiki will revert to English. You'll also have an option to choose a language different from that used in your browser.

The translation is performed by the Perl standard internationalization framework. If you want to contribute a new language, it would be most welcome: See TWiki:Codev.TWikiLocalization for instructions on how to help.

New options on the editing screen

You will notice a couple of changes to the editing screen; first, there is no switch for releasing the edit lock any more (if locks are enabled, they are always released when an edit finishes). You will also notice a new "force new revision" checkbox. TWiki normally doesn't add a new revision if the same user re-edits a topic within a certain time period; this checkbox allows you to force TWiki to add a revision for every change, regardless of how small it is.

Change notification

You now have much more control over which topics in a web you are interested in changes to. You can choose to receive notifications about topics selected by name, or by their parent relationship. See MailerContrib^? for more details.

Parameterised Includes

%INCLUDE{ "BugList" FAVOURITE="Damsel Flies" }%

My favourite bugs are %FAVOURITE%

My favourite bugs are Damsel Flies

Named Section Include

%INCLUDE tag allows to include only a named section of the included topic. These sections are defined in the included topic using the %SECTION% and %ENDSECTION% tags. For example, if the included topic has:

---+ News

---++ IT News
All news related to IT.
%SECTION{"itnews"}%
   * 2005-10-02 Final deployment of Dakar
   * 2005-10-01 Moving platform to Dakar
%ENDSECTION{"itnews"}%

Using %INCLUDE{ "AllNews" section="itnews" }% will produce:

   * 2005-10-02 Final deployment of Dakar
   * 2005-10-01 Moving platform to Dakar

This syntax also allows for nested sections. For example, given the following topic:

%SECTION{"outer"}%
   * Top Outer Text
%SECTION{"inner"}%
   * Inner Text
%ENDSECTION{"inner"}%
   * Top Outer Text
%ENDSECTION{"outer"}%

Using %INCLUDE{"SampleTopic" section="outer"}% will produce:

   * Top Outer Text
   * Inner Text
   * Top Outer Text

And %INCLUDE{"SampleTopic" section="inner"}% will produce:

   * Inner Text

In the same line, overlapped sections are allowed.

Notes for TWikiAdmins and WikiMasters

Security

Dakar Release introduces the use of 'safe pipes' to prevent any malicious request from executing code on the server. This strategy stops any of the known attacks dead in its tracks. The Dakar codebase has not been impacted by any of the recent security advisories in any way. Several public sites have been running Dakar code for some months now, and to the best of our knowledge none has been hacked.

CPAN Requirements

CPAN:Error and CPAN:Algorithm::Diff are now considered TWiki prerequisites, and are not shipped with the distribution itself anymore. CPAN:Text::Diff is a prerequisite for the UpgradeTWiki^? script only. CPAN:URI is a prerequisite for configure. Other new prerequisites are CPAN:CGI::Session and CPAN:CGI::Cookie, if you want to take advantage of the new session support.

If you want user interface internationalization support, CPAN:Locale::Maketext::Lexicon and CPAN:Encode (on perl 5.8's core) are required, as well as perl 5.8 or higher. See TWiki:Codev.TWikiInternationalization for details on TWiki internationalization support.

Installation and configuration

The installation and configuration processes has been significantly simplified.

setlib.cfg

TWiki.cfg

testenv / configure

The configure script can be used like the old testenv for public review of the configuration of the site. Saving from the interface is password-protected, using a password set in the configuration files, so to ordinary users configure just looks like a posh version of testenv. If you want to hide your configuration from public view, you can restrict access to the script webserver access controls (Apache users see the Apache documentation on the 'require' directive for more infomation on how to do this).

configure optional features

New optional features include {AutoAttachPubFiles} and {EnableHierarchicalWebs}. Both are switched off by default but can be enabled in configure.

Preferences

There have been some significant changes to the handling of preferences, aimed at making them more logical, consistent and easy to use.

Preferences are now evaluated in the following order:

1. Topic
2. Web
3. Session
4. User
5. Local Site (as set in {LocalSitePreferences})
6. Default (as set in {SitePrefsTopic})

Note that a user can still dictate preference values that can't be overridden, but they will now need to set those preferences as FINALPREFERENCES.

Permissions controls are not affected by this change.

Changes to the evaluation order

Preferences Plugin

The following standard preferences have been removed: MAILTHISTOPIC, MAILTHISTOPICTEXT, TOPICURL, READTOPICPREFS, TOPICOVERRIDESUSER (click on the name to search for occurrences on this site). If they are in use on your site, you can restore them to their Cairo settings by simply cutting and pasting the old definitions.

{SitePrefsTopicName} (was Main.TWikiPreferences)

Customized site preferences can now be saved in the {LocalSitePreferences} topic which will override settings in {SitePrefsTopicName}. This simplifies upgrades as you can overwrite the {SitePrefsTopicName} topic and gain any new or updated preference settings without losing your local customizations.

FAVICON

favicon.ico is a small graphic that can appear in a variety of places in the browser: the titlebar, the taskbar, the address bar, bookmarks/favourites, and tabs. Each web browser has a unique user interface, and as a result uses the Favicon in different ways. Most browsers display it in most of the locations listed.

Out of the box, TWiki is configured to easily customise the favicon.ico for each web. To switch to a new favicon.ico, upload it to the desired web's WebPreferences.

      * Set FAVICON = %PUBURL%/%WEB%/WebPreferences/favicon.ico

To provide a single, site-wide favicon.ico, hardcode a specific web, for example:

      * Set FAVICON = %PUBURL%/%TWIKIWEB%/WebPreferences/favicon.ico

FORCENEWREVISIONCHECKBOX

Normally, if you make another edit within a one hour period (was $editLockTime=in =lib/TWiki.cfg, now {ReplaceIfEditedAgainWithin}), TWiki will fold together your changes. This is often the "right thing to do", as it can reduce the visual clutter of diffs.

The "Force New Revision" checkbox is a way to force it to create a separate revision each time you save.

The TWiki.TWikiPreferences variable FORCENEWREVISIONCHECKBOX controls whether this is checked by default or not.

On a related note, you can force every save to be a new revision number by editing lib/TWiki.cfg and setting {ReplaceIfEditedAgainWithin} to 0.

NOTE: Although this feature is being introduced in this release, it is also being deprecated at the same time. TWiki:Codev.EdinburghRelease is planned to provide the ability to elide revisions at the GUI level, rather than the Store level, thus obviating the need for this stopgap measure.

WEBLOGONAME, WEBLOGOIMG, WEBLOGOURL, WEBLOGOALT

Each web can have its own customised logo. The simplest way is to upload a logo.png to a web's WebPreferences, and it will appear in the top-left corner.

To change the logo's filename, set the WEBLOGONAME variable. You'll especially need to do this if you use .jpg or .gif for the logo file format:

      * Set WEBLOGONAME = MyLogo.jpg

RSS feeds have also been enhanced to use the web-specific logo.

If you don't want to have custom logos on a per-web basic, but instead want to use a single, site-wide logo, hardcode a specific web in the WEBLOGOURL variable. For example:

      * Set WEBLOGOURL = %PUBURL%/%MAINWEB%/WebPreferences/logo.png

WIKILOGOIMG, WIKILOGOURL, WIKILOGOALT

These variables are now more closely associated with WIKITOOLNAME. If you change WIKITOOLNAME, you'll probably want to change these variables, too. WIKILOGOIMG, WIKILOGOURL, WIKILOGOALT, and WIKITOOLNAME are now used more consistently together.

Final Preferences

The FINALPREFERENCES setting prevents particular preference settings from being over-ridden at a lower level. The hierarchy of how FINALPREFERENCES settings are applied has been clarified/formalized as reflected in the following chart:

By default, the site level FINALPREFERENCES are set in %MAINWEB%.TWikiPreferences so as not to conflict with preference settings in that topic.

mod_perl support improvements

There is still an issue with the @INC path in mod_perl, that mainly impacts plugins that lazy-load modules. You should use the PerlSetEnv directive that mod_perl provides to make sure that your TWiki lib directory is permanently on the path, if you are using mod_perl.

Plugins

%INSTALLEDPLUGINS% and %DISABLEDPLUGINS% are no longer supported in TWikiPreferences. If you have set %INSTALLEDPLUGINS% in TWikiPreferences, you need to move that setting into the {PluginsOrder} configuration key, using the configure interface. To disable plugins, uncheck them in the configure interface, and save the changes.

Logins, Logouts, Sessions and Passwords

TWiki:Plugins.SessionPlugin and TWiki:Plugins.AuthPagePlugin have been integrated into the core. TWiki now supports cookied sessions, in the context of a much improved authentication architecture. The setup for authentication is now much simpler, and for most sites can be done entirely from the configure interface. There are some incompatibilities with TWiki::Plugins.SessionPlugin, with resepect to the in-line variables. See TWikiUserAuthentication in the release for full details of how authentication works now. TWiki also now supports the concept of pluggable password managers, making the integration of corporate authentication services much simpler.

Protections

The evaluation of protections has been re-worked to make it more naturally understandable, and also fill a number of holes in the protection scheme, These holes meant that it was relatively easy to deny access to a topic, but rather difficult to subsequently restore access without either compromising other topics, or compromising old revisions.

When deciding whether to grant access, TWiki now evaluates the following rules in order (read from the top of the list; if the logic arrives at PERMITTED or DENIED that applies immediately and no more rules are applied). You need to read the rules bearing in mind that VIEW, CHANGE and RENAME access may be granted/denied separately.

1. If the user is a super-user
   • access is PERMITTED.
2. If DENYTOPIC is set to a list of wikinames
   • people in the list will be DENIED.
3. If DENYTOPIC is set to empty ( i.e. Set DENYTOPIC = )
   • access is PERMITTED i.e. no-one is denied access to this topic
4. If ALLOWTOPIC is set
   1. people in the list are PERMITTED
   2. everyone else is DENIED
5. If DENYWEB is set to a list of wikiname
   • people in the list are DENIED access
6. If ALLOWWEB is set to a list of wikinames
   • people in the list will be PERMITTED
   • everyone else will be DENIED
7. If you got this far, access is PERMITTED

The major impact of this change is that WebPreferences topics shipped with earlier releases of TWiki will have excessively restrictive controls, as the default settings were:

• Set DENYWEBVIEW =
• Set ALLOWWEBVIEW =
  • This will now deny view access to everyone not in the list (i.e. everyone except admins)
• Set DENYWEBCHANGE =
• Set ALLOWWEBCHANGE =
  • This will now deny change access to everyone not in the list (i.e. everyone except admins)
• Set DENYWEBRENAME =
• Set ALLOWWEBRENAME =
  • This will now deny rename access to everyone not in the list (i.e. everyone except admins)
• Set ALLOWTOPICCHANGE =
  • This will now deny change access to everyone not in the list (i.e. everyone except admins)
• Set ALLOWTOPICRENAME = Main.TWikiAdminGroup

The standard webs shipped with this release have these settings disabled. However you are likely to have inherited the old default settings in your user webs. The easiest way to deal with this is to simply insert a # sign in these settings; for example:

• #Set DENYWEBVIEW =
• #Set ALLOWWEBVIEW =
• etc

SitePermissions

This topic gives you a quick view of the permissions on each web - i.e. it aggregates ALLOWWEB* etc. into a handy table. It is also installed on TWiki.org as TWiki:TWiki.SitePermissions but is not in Cairo.

Frustrating Robots

New User Registration

The new user registration process has been extensively reworked to improve usability and security of the registration process.

1. FirstName and LastName are taken separately and tabulated. See http://wiki.conceptmapping.org/People/UserListByLocation
2. Sends a email that the user has to respond to; registration data is staged until this is done
3. Populates a form in the user topic, if there is one in the NewUserTemplate. If there is no form, then bullets are used, as in pre-Dakar TWiki.
4. Sends different emails to the WIKIWEBMASTER and the USER, with the user's containing the password unless {HidePasswdInRegistration} is set.
5. The templates/view.pattern.tmpl template is rearranged to show the form on the top LHS, any pictures on the top RHS (uses TWiki:Plugins.ImageGalleryPlugin if installed), and any fields not in the form (e.g. comment) in the content in the topic.
6. There is now a mechanism for bulk registration, with callbacks for augmentation (e.g. from other sources).

1. Absorbed ResetPasswordByEmail^? .
2. Obsoleted InstallPassword^?

Change notification support

SiteChanges

This topic mimics TWiki.org's TWiki:Codev:WebChangesForAllWebs - i.e. it shows a WebChanges view across a whole site. It's name was chosen to parallel TWiki.SiteMap; at some point you can expect the arrival of TWiki.SiteStatistics too.

What's a Web

Disable <script> tags

Notes for TWikiApplication Developers

Space/Tab conversion

Previous TWiki versions would automatically convert three spaces at the start of lines to a tab when the topic was saved. This meant that the saved topic was not the same as the edited topic, which could result in considerable confusion. This conversion has been disabled, and the saved topic is now exactly what you see in the editor. One impact of this change is that any add-on scripts you may have developed that rely on bulleted list lines starting with a tab will no longer work. They must be adapted to treat groups of three spaces and single tabs as equivalent.

Evaluation order of TWikiVariables

In previous TWiki versions the evaluation order of %TAG%s depended on where they were expanded in the code. The parser was somewhat crude, and could easily be confused when embedded variables (variables embedded in the parameters of other variables) were used.

The parser has been replaced in Dakar with a deterministic variable parser with predictable behaviour. Specifically, variables are now always evaluated left to right and inside out. For example, consider %TAG2{ "%TAG1{ "%TAG0{ "params" }%" }%" }% %TAG3%. Previously, the expansion order would have depended on the order of expressions in the code, so the expansion may have proceeded TAG3 - TAG0 - TAG2 - TAG1. If you were lucky, this was the intended order. In Dakar, the order is now guranteed to be TAG0 - TAG1 - TAG2 - TAG3 (i.e. inside out and left to right).

The main impact of this is that some TWikiApplications^? may cease to work if they have been written to take advantage of the old chaotic order. There is no way to predict which will work and which will fail, so you will have to deal with this on a case-by-case basis. In most cases TWikiApplication authors will have worked hard to do the "sensible thing" so instances of this problem should be rare.

Note that because the TWiki spec allows double quotes within double-quoted strings in certain variable parameters it has been impossible to make the parser 100% deterministic. There may still be pathological cases where the parser may fail. In these cases, consider how open and close curly brackets are matched up.

Skin search paths

New in Dakar release is support for skin search paths which allow you to specify any number of skins, instead of the default single skin. If a template can't be found for the first skin in the path, then the second skin is used, and so on until the default template is used as a last resort. See TWikiSkins for more information.

Encoding of form-field values

<script> tags in topics

In recognition of security concerns around <script> tags, the administrator has the choice whether to allow users to add script to topics or not. Check the setting of {AllowInlineScript} in configure to see if it is allowed on your site. If not, script sections will simply disappear from topics.

ALL_VARIABLES

You can use %ALL_VARIABLES% in a topic to get a dump of all variables set in that context. Invaluable for debugging those tricksy TWikiApplications!

IF

The new %IF()% variable defines simple conditional statements that are evaluated at view time. This allows you to include content conditionally based on environmental factors. See ViewDEVELOP^? :TWiki.IfStatements for more information on usage.

New $count(reg-exp) variable in FormattedSearch

Notes for Skin Developers

Supporting web names that are WikiWords

Skins should be upgraded if they have standalone %WEB% variables; only standalone %WEB% text that potentially could be turned into a link (because of a WikiWord) needs to be escaped. Same for %MAINWEB% and %TWIKIWEB%.

Examples:

• %WEB% -- needs to be escaped with <nop>%WEB%
• (%WEB%) -- needs to be escaped because of parenthesis
• "%WEB%" -- no need to escape, does not get linked
• <b>%WEB%</b> -- no need to escape, does not get linked
• %WEB%.%TOPIC% -- no need, is a Web.TopicName
• (%WEB%.%TOPIC%) -- no need, is a Web.TopicName

Basically, any prefix other then space and parenthesis needs to be looked at. %WEB% in a %SEARCH% should not be escaped.

Deprecation of %EDITTOPIC%

Dakar release includes new support for "context if" parameters to the %TMPL:P construct. See TWikiTemplates for details. The default templates shipped with Dakar have been modified to use this support. %EDITTOPIC has been deprecated, though it is still available as a simple edit link, defined in TWikiPreferences. Skin authors are strongly recommended to replace this link with context-if conditionals.

Template Parameters

HTTP and HTTPS

QUERYSTRING

Notes for Plugin Developers

Internals

A lot of the TWiki internals have changed. As a result, plugins that bypass the TWiki::Func API and call core functions directly are unlikely to work.

The restructuring of the code internals is such that there are no 1:1 equivalents for the old core functions. Only the TWiki::Func API is guaranteed to work.

You should convert your plugins to call the TWiki::Func API. If you have called unpublished functions that have no equivalent in TWiki::Func, then you may still be able to call the function via the TWiki "session" object, $TWiki::Plugins::SESSION. See the implementation of the TWiki::Func module for ideas on how to do this. However calling internals is not recommended, even using this new mechanism, as they are liable to change without prior notice.

Extensions to the Func API

• The TWiki::Func API has been extended to expose a number of new core functions. Review TWikiFuncDotPm for details.
• The TWiki::Meta API, which was previously for internal core use only, has now been exposed and may be used in plugins. See TWikiMetaDotPm for full details.
• Plugins which observed the existing published API (TWiki::Func) should continue to work.

Public contributions, tested on Dakar

Most contributions have had to be modified to work with Dakar. These modifications have been required to remove abuses of the published APIs.

Contributions that don't conform to standards sufficiently to be compiled

Contributions that fail initPlugin

Contributions that can't be loaded for testing because of missing perl components

Contributions that load and don't fail, but don't work as advertised

Contributions that load harmlessly, but cannot be fully tested due to lack of tests, or lack of external dependencies.

Contributions that are known to work with Dakar

The jury is still out on these