Single Branch Plugin Development

This MAIN Branch has moved to http://svn.twiki.org/svn/twiki/trunk/ in 17 Feb 2008

Note: this topic has been heavily refactored for the discussion on moving to individual extensions branches

existing TWiki Subversion Repository Layout

There are several different developer roles in the TWiki project:

• Core developers
• Active developers/maintainers of several extensions (plugins, TWikiApplications, Contribs etc)
• Casual developers/maintainers of one extension
• Throw-it-over-the-wall extension developers

We want to make it as easy as possible for anyone (especially casual extension developers) to find their way around in the subversion repository, without having to RTFM. This means we need a really simple and obvious scheme for branch naming, so that it is clear where to check stuff in. However, to support more frequent contributors, we also need a branch per release for preparation and maintenance. So we use the following scheme:

• a MAIN branch for working in the latest code. Extensions are developed in the twikiplugins directory of this branch.
• a branch per release (major, minor) e.g. TWikiRelease04x02. Once released, these branches are only used for development of patches to that release.
• One tag per release (major, minor and patch). These tags are completely read only. No one can be allowed to check in. Ever.

Example of the Existing layout (at the time of the 4.2.0 release)

\
| - branches
       | - MAIN
             | - twikiplugins
                       | - GetYourPlugin
       | - TWikiRelease04x02
             | - twikiplugins
                       | - GetYourPlugin
| - tags
       | - TWikiRelease04x00x00/
       | - TWikiRelease04x00x01/
       | - (many, many, many more)

This model has served us well - up to a point. But:

• From code freeze through to release is supposed to be a short time, but for the last two releases it has been an extended period. The build system builds from a single branch, so developers working on "default" extensions (those bundled with a standard TWiki release) are faced with having to check in changes to two branches - the release branch and MAIN. This has proven to be error prone (at least when I try to do it).
• The twikiplugins directory is an artifact of the build process. It is not part of the release, it is just a place to "hang" extensions. It is difficult to explain, and difficult to use unless you are reasonably adept with the infrastructure tools (such as build.pl)
• Anyone who wants to check out a copy of TWiki for development is forced to check out all the extensions as well, which is a long, slow process.
• It is almost impossible to specify a specific version of an extension for inclusion in a release; you have to take whatever is at the head of the release branch.

The accepted Proposal

An alternative, more user-friendly, approach would be to recognise that extensions are developments in their own right, and are not slaves of the TWiki release process. This recognition leads inevitably to the idea that each extension should inhabit its own place alongside the core. For example,

\
| - trunk
      | - core
           |- lib
               | - TWiki
      | - GetYourPlugin
           | - lib
               | - TWiki
| - branches
       | - TWikiRelease04x02
           | - core
                 | - lib
                      | - TWiki
           | - TWikiUserMappingContrib (and all other Contribs that are part of the release)
                 | - lib
                     | - TWiki
           | - GetYourPlugin (optional)
                 | - lib
                     | - TWiki

This has several advantages:

• Much easier for new developers to understand
• Cleaner and simpler - consistent with Subversion Red Book
• build.pl for releases can be much simplified
• Supports branching and tagging of all components
• Extensions can easily be branched, if we need it
• Subversion access controls are easier to set up and maintain
• The core is maintained without explicit dependencies on extensions, opening the door to easier TWikiFor releases
• Extension authors have greater control over their own release cycles

The following changes to infrastructure would be required:

• Rename 'MAIN' to 'core', and move it from 'branches' into 'trunk'
• Move extensions from twikiplugins/ into trunk/
• Adjust access controls to grant extensions authors appropriate rights
• Modify build scripts as appropriate

-- CrawfordCurrie, KennethLavrsen, RafaelAlvarez, SvenDowideit, PeterThoeny

Comments on first draft (rev 91)

I agree 100%.

Extensions need to be ONE place.

One small detail you also mention. How do we practically limit access to extentions and not the core? And should be just give general access? If people have access to other people's extensions why not also the core? It is not like we have too many people helping with core maintenance.

-- KennethLavrsen - 22 Jan 2008

There must be a way for plugin authors to make "tags" for each release, too. That way, if an bug is fixed in an old version of a plugin, it can be fixed without forcing an upgrade (or forcing the plugin author to maintain backward compatibility). A small modification to the proposal can achieve this:

\
| - branches
       | - MAIN
             | - lib
                  | - TWiki
       | - GetYourPlugin
             | - HEAD
                     | - lib
                          | - TWiki
             | - releases
                   |- GetYourPlugin10
                   |- GetYourPlugin20
       | - TWikiRelease04x02

Changed needed to the infrastructure:

• Modify build scripts to support specification of a specific rev or a particular release of an extension to include (default HEAD)

Also, if we put the plugins in a plugins branch, it could be helpful for those of us who actually checkout all the plugins, and also will keep the branches directory "clean".

-- RafaelAlvarez - 22 Jan 2008

That looks like a really difficult thing to manage.

And SVN allows you to checkout any version you like. All you need to know is the SVN number and that is easy to find with svn log. There is no need to tag all those plugins. Normal users will never know to get versions from SVN anyway.

-- KennethLavrsen - 22 Jan 2008

While Raf's layout is one of the more common svn layouts - treating Plugins as the same configuration management complexity really does not thrill me. In fact, Crawfords suggestion doesn't thrill me either :). One thing that bothers me about it, is that to check out TWiki MAIN, and all the plugins, I am basically forced to checkout all of the branches of TWiki (maybe thats a good thing, but I am running out of disk space).

How about we do something frighteningly simple:

\
| - branches
       | - MAIN
             | - lib
                  | - TWiki
       | - TWikiRelease04x02
| - twikiplugins
       | - GetYourPlugin
             | - lib
                  | - TWiki

Thus, to work on aset of plugin with TWiki4.1.1, we can checkout the TWikiRelease04x01x01 tag, and the twikiplugins dir (into the 411 checkout) and use pseudo-install like we always have.

IF it comes to pass that we do need branching of plugins, adding a twikipluginsbranches with the added complexities can be done without making life more complex for the majority.

While we're at it, can we delete the 'trunk' its a confusing relic of an older time.

-- SvenDowideit - 22 Jan 2008

Sven is right. It will be a pain to checkout each plugin by themselves. And too much to checkout all branches.

-- KennethLavrsen - 22 Jan 2008

Sven's proposal looks very clean. It also makes it easy to have one access control setting for core, and one for all plugins, which as proven to work quite well.

-- PeterThoeny - 23 Jan 2008

Yes, agreed, that is much cleaner. Refactored the proposal to reflect it.

-- CrawfordCurrie - 23 Jan 2008

Comments on second draft (rev 98)

Sven, I'm not quite clear if you are expecting us to tag extensions releases, as Raf suggests?

On that note, it's impractical to maintain some extensions from a single set of sources. For example, a user mapper. Or even a skin may want to leverage facilities only available in the latest and greatest. We need to be able support different sources for different TWiki versions. Ideas? Maybe:

\
| - branches
       | - MAIN
             | - lib
                  | - TWiki
       | - TWikiRelease04x02
| - twikiplugins
       | - GetYourPlugin
             | - lib
                 | - TWiki
             | - releases (optional)
                 | - 04x01x02
                     | - lib
                          | - TWiki

which is very similar to Raf's proposal. It would make it easy for the TWiki release manager (just !include twikiplugins/CommentPlugin/releases/04x03 in MANIFEST), and the optional nature of the releases directory keeps it simple for most plugins authors, while still allowing more mature authors scope to perform better version control.

-- CrawfordCurrie - 23 Jan 2008

I think we should avoid tagging and branching of extensions, right up until the point that it becomes impossible to continue without. Certainly, I don't think the suggestion you have above is worth it if its just for the convenience of the ReleaseManager.

For example, If a branch becomes necessary, we could then create a /branches/twikiplugins/TWikiRelease04x01x02/GetYourUnpopularPlugin. This is much more consistent with the svn layout we have for the Core (lets not use both svn layouts in the one repository).

-- SvenDowideit - 23 Jan 2008

It's not for the convenience of the release manager, it's mainly for me. I have several extensions that have code specific to individual TWiki versions. I have others where the extended Func API is used, and they won't work on "old" TWikis. I'd like to be able to check out a plugin and get all those versions in one hit, though that isn't essential.

Your suggestion:

\
| - branches
       | - MAIN
             | - lib
                  | - TWiki
       | - TWikiRelease04x02
       | - extensions
             | - TWikiRelease04x02
                   | - GetYourPlugin (optional)
                       | - lib
                            | - TWiki
| - extensions
       | - GetYourPlugin
             | - lib
                 | - TWiki

does my head in a bit, but would also be OK with me (I renamed twikiplugins to extensions because it is so much more than just plugins now)

-- CrawfordCurrie - 23 Jan 2008

if we're going to rename twikiplugins, we may as well move /branches/MAIN to /trunk

-- SvenDowideit - 23 Jan 2008

That's actually a good point. Are we at risk of making this far more complex than it needs to be? Consider:

\
| - trunk
      | - core
           |- lib
               | - TWiki
      | - GetYourPlugin
           | - lib
               | - TWiki
| - branches
       | - TWikiRelease04x02
           | - core
                 | - lib
                      | - TWiki
           | - GetYourPlugin (optional)
                 | - lib
                     | - TWiki

This has several advantages:

• Cleaner and simpler
• build.pl for releases can be much simplified
• supports branching and tagging of all components

-- CrawfordCurrie - 24 Jan 2008

Comments on third draft (rev 109)

yep. that is how it should be. I'll make it so if there is agreement from a few more people

-- SvenDowideit - 24 Jan 2008

Moving to trunk? No worries. If this can help a newcomer developer I am all for it.

It will be smart to move all the default plugins to the TWikiReleaseXXX branch. Ie. not making it optional for those. This way we can stabilize those and not impose a feature freeze on them when the release branch is created for first beta.

-- KennethLavrsen - 24 Jan 2008

Kenneth, correct. If its in a release, it needs to be tagged and branched.

-- SvenDowideit - 25 Jan 2008

Sounds like we have agreement. Last chance, community (esp. Rafael & Peter) - comment or accept the consequences!

-- CrawfordCurrie - 25 Jan 2008

The last layout from Crawford looks good. It addressed my two concerns (Checkout all plugins in one pass, and plugin branching). Go ahead

-- RafaelAlvarez - 25 Jan 2008

Ok, I will look into making it so next weekend - Probably will happen that monday (the first Monady of feburary 2008) - I'll send out some warning emails mid this week.

-- SvenDowideit - 27 Jan 2008

I would suggest that it may be time to use a modern distributed version control system for development. SVN is quite nice for enforcing release processes with its centralized model, and have branches/tag per major/minor/patch release. But for actual development of features, a branch per feature seem the natural solution but SVN can be cumbersome to work with. The two major open source Distributed Source Control systems are:

• Git famous for being made by Linus Torvalds and used for Linux
• Mercurial a kind of fork of Git, simpler and less linux-centric (can be used on windows)

I would recommend listening to Linus make his point in the Video http://www.youtube.com/watch?v=4XpnKHJAok8 then just following the Mercurial tutorial. I am starting up on mercurial, and it really seems the right way to handle the one-branch-per-feature development mode

-- ColasNahaboo - 27 Jan 2008

I had a deadline, so the change will now occur before the end of Feburary2008, and I will send out an email a few days before the work happens, and again the day of the change - describing the detailed plan of execution. One effect will be that for a few hours, TWiki svn will be either read only, or offline.

To be clear, I'm not against the proposal, I just want to make sure that it acheives the goals that people think it will.

This proposal does not change the realities of a release branch - whenever we're close to a release, we will be branching both the core, and any extensions that will be in that release, and the major development focus will move to that branch. Extensions and new features not destined for that release will thus continue on the trunk (ie, no major change from how we did 4.2.0, its just that there is less mess in the branch (we could just as easily svn rm non-relevant plugins from the branch.)).

The main concerns are day to day, non release branch development - If someone wants to develop an extension they could do so without checking out very much - modern best practice would suggest checking out BuildContrib beside the TWiki release unzip that they are targeting, making a new Contrib directory which they can then svn import and commit...

Thus, the 'simple' layout would be 'similar' to the svn

\
| - WorkingTWikiRelease4x1x1 (made by unzipping)
      |- lib
           | - TWiki
| - WorkingTWikiRelease4x1x2 (made by unzipping)
      |- lib
           | - TWiki
| - WorkingTWikiRelease4x2x0 (made by unzipping)
      |- lib
           | - TWiki
| - YourNewPlugin
      | - lib
               | - TWiki
| - BuildContrib
      | - lib
               | - TWiki

You can thus share BuildContrib with multiple release TWikis, and pseudo-install YourNewPlugin into any of them.... though BuildContrib will need much more cleverness, and you would need to specify a destination TWiki install BuildContrib/pseudo-install.pl YourNewPlugin WorkingTWikiRelease4x1x2/lib/LocalSite.cfg for eg.. Note, this is already possible with the existing layout - I do it all the time by just copying the most recent pseudo-install into an old release's twiki dir, and then checking out the parts of twikiplugins that I need

For Core developers, little changes - its best to check out the entire svn/trunk so as to be able to quickly test the results of changes on several plugins... We'll need to extend pseudo-install to have a development target similar to the default we have now, and perhaps even a BuildContrib svn bootstrapper - which can check out a TWikiFor (ie core plus extenstions) and pseusdo-install them (possibly even attempting a pre-install. And Thus perhaps we get TWikiForTWikiDeveopment.

On reflection, this proposal seems to change mostly the names - to the standard expected by svn red-book - and that's a good thing.

-- SvenDowideit - 05 Feb 2008

Right. you would need to specify a destination TWiki install - you may not have noticed it, but $ENV{TWIKI_HOME} is designed for that specific purpose. The unit tests acknowledge it already as well.

Your example is spot on, otherwise.

-- CrawfordCurrie - 05 Feb 2008

I detest build processes that hope someone has remembered to set or update environment variables. Especially as I almost always work on several versions of TWiki at one time (from the same terminal windows). So long as I can not set them, and specify what I intend at the time I intend it, I'm happy for you (and most people ) to have a convenient short cut.

Excellent - looks like we're good to go - I'll compose a plan to execute, and email out some documentation on howto transform your existing checkout to the new form.

Execution plan is tracked at TWikibug:Item5341

-- SvenDowideit - 05 Feb 2008

in 17 Feb 2008, TWiki's core and extensions development moved to http://svn.twiki.org/svn/twiki/trunk/ - see SingleBranchPluginDevelopment and RefactorSubversion fo rthe discussion, and TWikibug:Item5341 for the implementation.

-- SvenDowideit - 17 Feb 2008