NOTE: THIS TOPIC IS IN TWO BITS; THE FIRST BIT DEALS WITH THE PRAGMATIC SHORT_TERM BUILD SOLUTION WORKED OUT BY SvenDowideit AND CrawfordCurrie SO WE CAN BUILD A RELEASE. THE SECOND PART DEALS WITH THE LONGER TERM BUILD INFRASTRUCTURE BEING BUILT BY WillNorris. IF YOU JUST WANT TO BUILD A TWikiFor BASED ON THE Dakar CODE, REFER TO SECTION 1 ONLY.

Part 1 Building DakarRelease

The build uses BuildContrib, that works using

1. a build.pl file, equivalent to a makefile
2. a MANIFEST that describes the contents of the build
3. a DEPENDENCIES file that lists the dependencies

Examples of all these files can be found throughout the twikiplugins repository. Read the comments in the BuildContrib code for more information on what you can/can't do.

Building the release packages

To build a zip and a tgz for Dakar, first checkout from SVN. Then:

cd tools
perl build.pl release

release is called the target of the build process. There are other targets, such as test, manifest, and even upload (which uploads the release to TWiki.org. In the wrong place, but hey, you can't have everything)

This is the simplest possible way to do a build. Of course the actual process is a little bit more involved, as tests have to be run. The full process flow can be found at BuildingARelease.

Bundling additional plugins

If you want to build something other than the generic TWikiFor, look in tools/MANIFEST. Add plugins/contribs to your specification at the end, following the pattern of the ones already there. They must be buildable by BuildContrib but it's easy to make them so.

To make a plugin buildable (e.g. PutThePlugin)

cd twikiplugins/PutThePlugin/lib/TWiki/Plugins
svn mkdir PutThePlugin # if it doesn't already exist
cd PutThePlugin
touch MANIFEST DEPENDENCIES
cat ../../../SmiliesPlugin/lib/TWiki/Plugins/SmiliesPlugin/build.pl | sed -e 's/Smilies/PutThe/g' > build.pl
svn add build.pl MANIFEST DEPENDENCIES
perl build.pl manifest > MANIFEST # to grab all the files
# check the content of MANIFEST to make sure it makes sense; remove any ,v files and any comments.
# ...and then, to make sure it works,
perl build.pl release 

If you want to bundle a Contrib, the process is the same except build.pl is held under the TWiki/Contrib/MyContrib dir. We haven't worked out any way to bundle skins or add-ons yet, though if someone wants to do that, it should be fairly easy to work out a process.

Part 2 Longer-term build support

This describes the build procedure used to create the generic TWikiFor which can be downloaded from http://twikiplugins.sourceforge.net/twiki.tar.bz2

Overview

Because TWiki is written in Perl, and the TWiki development repository is very similar to the distribution, the amount of work a build has to do is very small. Mostly it is to do with separating out the shipped pieces from the development-only pieces, building documentation that is derived from comments in the code, and automatically extracting differences in TWiki topics between the build and the last official distribution.

The build process for TWiki works by assembling a local image of the distribution on your local machine, and then zipping and uploading this image to the distribution site.

The following instructions are for building and uploading a bzip2 package.

Generic TWikiFor build requirements

• The scripts are targeted for Linux, though should work on most Unices.
• Perl 5.8
• The Apache htpasswd program (to generate TWiki.TWikiGuest/guest .htpasswd)
• xsltproc (to generate CHANGELOG from SVN)
• svn (subversion client)
• make (Only GNU make has been tested)
• CPAN:LWP CPAN:Number::Compare CPAN:Text::Glob CPAN:File::Slurp CPAN:File::Find::Rule CPAN:File::Slurp::Tree CPAN:File::Spec::Functions CPAN:Getopt::Long CPAN:Pod::Parser CPAN:LWP::UserAgent CPAN:LWP::UserAgent::TWiki::TWikiGuest
• ssh keys (optional, though highly recommended for publishing the resulting TWikiFor)

(Note that build times have increased by an order of magnitude since ConvertTWikiPluginsCvsToSvn, which means I'll have to do some profiling of the various programs at some point, although I do already know a lot of the time is related to performing remote operations on a large repository)

Build account prerequisites

Setup for the build process

Build user

BestPractices say that you should do builds as a special build user. This is recommended, though not essential. The scripts that follow assume that you are building as user twikibuilder. This user should have the following environment variables set:

Variable	Description	Example
TWIKIDEV	Root directory where you are doing the build. Branches must be checked out under this directory.	~/twiki
BRANCH	Subversion branch name that we are publishing	MAIN
PUBLISH_TO	Publishing target URL. WIll be written to using LWP	sourceforge_user@twikiplugins.sourceforge.net:/home/groups/t/tw/twikiplugins/htdocs

Example setup to publish the MAIN branch to a user area on sourceforge.net

export TWIKIDEV=~/twiki export BRANCH=MAIN export PUBLISH_TO=sfuser@twikiplugins.sourceforge.net:/home/groups/t/tw/twikiplugins/htdocs

(you can place this in your ~/.profile file)

You should also generate ssh keys for secure and passwordless uploading:

ssh-keygen -t dsa -b 1024

upload ~/.ssh/id_dsa.pub key to the computer you are going to publish to. There are directions for sourceforge here (TODO: find references to standard unix commands)

Checkout

The build requires a checkout of the source for the given $BRANCH. For example,

[ -d $TWIKIDEV/$BRANCH ] || mkdir -p $TWIKIDEV/$BRANCH cd $TWIKIDEV/$BRANCH svn --username TWikiBuilder checkout http://svn.twiki.org/svn/twiki/branches/$BRANCH .

The build process will create some files in this checkout area, but will not overwrite any files in the checkout area, except those it generates itself.

CPAN local minimirror

A local minimirror of CPAN is required for the CPAN prerequisites for the build scripts, and for installing, testing, and bundling the CPAN prerequisites for the target TWikiFor.

You can create and populate one as follows.

cd $TWIKIDEV/$BRANCH/twikiplugins/TWikiInstallerContrib/lib/TWiki/Contrib/TWikiInstallerContrib/cpan ./mirror-cpan.pl --cpan

Install CPAN prerequisites

these are build script dependencies; the resultant TWiki package does not depend on these modules.

For building the TWikiCore

cd $TWIKIDEV/$BRANCH/twikiplugins/TWikiInstallerContrib/lib/TWiki/Contrib/TWikiInstallerContrib/cpan ./install-cpan.pl -f -mirror=./MIRROR/MINICPAN -baselibdir=$TWIKIDEV/CPAN Number::Compare Text::Glob File::Slurp File::Find::Rule File::Slurp::Tree File::Spec::Functions Getopt::Long Pod::Parser LWP::UserAgent LWP::UserAgent::TWiki::TWikiGuest </dev/null

For the TWikiFor build

cd $TWIKIDEV/$BRANCH/twikiplugins/TWikiInstallerContrib/lib/TWiki/Contrib/TWikiInstallerContrib/cpan ./install-cpan.pl -f -mirror=./MIRROR/MINICPAN -baselibdir=$TWIKIDEV/CPAN XML::NamespaceSupport XML::SAX XML::Simple </dev/null

To be able to run the unit tests

cd $TWIKIDEV/$BRANCH/twikiplugins/TWikiInstallerContrib/lib/TWiki/Contrib/TWikiInstallerContrib/cpan ./install-cpan.pl -f -mirror=./MIRROR/MINICPAN -baselibdir=$TWIKIDEV/CPAN Error Class::Inner Devel::Symdump Test::Unit Apache::Htpasswd HTML::TreeBuilder </dev/null

To run a tinderbox

cd $TWIKIDEV/$BRANCH/twikiplugins/TWikiInstallerContrib/lib/TWiki/Contrib/TWikiInstallerContrib/cpan ./install-cpan.pl -f -mirror=./MIRROR/MINICPAN -baselibdir=$TWIKIDEV/CPAN XML::Parser Test::Manifest XML::RSS </dev/null

Directory Layout

Layout of a build environment.

$TWIKIDEV/

CPAN/ - locally-installed CPAN modules used by build and test processes

bin/

lib/

man/

.cpan/ - local CPAN build and cache directory

$BRANCH/

twikiplugins/TWikiInstallerContrib/lib/TWiki/Contrib/TWikiInstallerContrib/

bin/

download-twiki-extensions.pl

install-remote-twiki.pl

svnrev.pl

calc-twiki-deps.pl

cpan/

install-cpan.pl

mirror-cpan.pl

MIRROR/ - local CPAN minimirrors (contains newest version (only) for each module)

MINICPAN/ (562MB - 31 jul 05) - minimirror of all CPAN modules except Acme:: and perldist modules

TWIKI/ (29MB - 31 jul 05) - another CPAN minimirror, of the modules used by TWiki and plugins (created from the MINICPAN minimirror)

downloads/

kernels/ - kernels assembled by build-twiki-kernel.pl (later, build-twiki-MANIFEST.pl)

plugins/ - downloaded from twiki.org which conform to attachment $PluginTopic.zip on TWiki:Plugins.$PluginTopic

addons/ - same as plugins

contribs/ - same as plugins

skins/ - not yet addressed

platform/

install_twiki.cgi - the actual installer, also html menu config for installation

pre-twiki.pl - installs local cpan libraries required by the installer

(next step is to merge this into install_twiki.cgi or MANIFEST-related scripts)

un-twiki.pl - uninstall the twiki installation

uninstall.pl - remove this whole package (deletes everything that was unpacked from twiki.tar.bz2)

config/ - the HTML for the GUI installer (needs to be reevaluated in the context of configure)

tools/

distro/

(downloaded or generated files and directories)

Each time you build:

Build the Core (no plugins)

First ensure the checkout is up-to-date:

cd $TWIKIDEV/$BRANCH svn update

Now run the actual build script to assemble the components of the release. The build process builds an image of the install in $KERNEL_OUTPUT_DIR.

KERNEL_OUTPUT_DIR=$TWIKIDEV/$BRANCH/twikiplugins/TWikiInstallerContrib/lib/TWiki/Contrib/TWikiInstallerContrib/downloads/kernels/ [ -d $KERNEL_OUTPUT_DIR ] || mkdir -p $KERNEL_OUTPUT_DIR cd $TWIKIDEV/$BRANCH/tools/distro time ./build-twiki-kernel.pl --notar --outputdir=$KERNEL_OUTPUT_DIR

Use ./build-twiki-kernel.pl --help for help on options

Bundle installer + core + plugins + contribs + CPAN

Required plugins and contribs are built from sources in the twikiplugins area. The generated packages are then installed to the release area. The release area is then bundled up for distribution. TODO: must run build.pl for plugins that are so configured. build.pl is not part of the shipped package, it is the build script for the plugin/contrib.

export MakeFor=twiki.org cd $TWIKIDEV/$BRANCH/twikiplugins/TWikiInstallerContrib/lib/TWiki/Contrib/TWikiInstallerContrib # MakeFor=twiki.org-`bin/svnrev.pl` time make distro

Publish the resultant TWikiFor

This will upload the final package (using scp) to $PUBLISH_TO.

cd $TWIKIDEV/$BRANCH/twikiplugins/TWikiInstallerContrib/lib/TWiki/Contrib/TWikiInstallerContrib make publish

Questions

1. How do I specify the options for my TWikiFor? Where do I list the plugins I want included?
2. Does the distributed package include tests?
   • well, it currently (and incorrectly) includes the TestCases web, this will be addressed in more detail when I finish documenting more TWikiFor and MANIFEST details. btw, what do you want it to do?
   • personally I do not want to ship tests, unless someone wants to identify a subset of tests that can be run without the test infrastructure (CPAN:Test::Unit and TestFixturePlugin)
   • ok, tests are no longer included
3. Is the distributed package signed?
   • No. This is discussed in ShipMd5Sums.

-- Contributors: WillNorris, CrawfordCurrie

Discussions

The #CPAN_local_minimirror section has useful information. Could someone copy/move relevant information to TWiki.HowToInstallCpanModules supplemental documentation so that the information is more readily available?

-- PeterThoeny - 18 Dec 2006

It is already done.

-- TWikiJanitor - 04 Sep 2008