RefactorSubversion

Sorry if I'm duplicating an existing topic; if so, I couldn't find it. I know this has been discussed before, but there isn't an open, trackable request that I can find, so.......

It's obvious to anyone who uses subversion that (A) checkouts are pretty huge these days and (B) there is no need to duplicate the twikiplugins area in every checkout. Extensions are versioned on MAIN, and there is arguably no need to version them on every branch and tag.

Rather than trying to capture a technical solution to this, it would be helpful to start with some requirements (usual anal interpretations of must and may):

1. The set of people permitted to check into the extensions area must be different to the set of people able to check into the core
2. A checkout of the core must not include non-core extensions
3. A core checkout may include default plugins
4. Individual extensions may be independently branchable/taggable
5. It must be possible to run a TWiki directly based on a SVN checkout, i.e. without going through an build/install cycle first (perl scripts may be used, but must be portable to Windows)
6. It must be possible to run unit tests on a SVN checkout.
7. It must be simple to branch and tag a TWiki version with the 'default' Extensions included in the branch/tag,

Any more?

-- Contributors: CrawfordCurrie, HaraldJoerg

Discussion

Yes, I remember this discussion from long time ago. Just of the records, IIRC, the main argument for having the plugins in the same "checkout" was to be easy to create a TWiki installation based on a SVN checkout.

IMO, this issue can be easily circumvented by having some nice scripts that performs the plugins checkout in the proper place for pseudo-install.pl to pick them up.

-- RafaelAlvarez - 28 Sep 2007

I fully agree. The huge checkouts have been a problem for me (disk size of virtual machine, checkout time)

other remarks incorporated in the requirements

-- HaraldJoerg - 28 Sep 2007

And I, as the current release builder, require

1. it must be possible and normal to branch and tag a TWiki version, and add the 'default' Extensions into that banch/tag, so that when we're in a beta phase like now, or, many months from now need to build a 4.2.5, that we're not upgrading some plugins in massive jumps.
   • I moved this into the requirements CC

When I'm building a patch release, I want to be sure that I'm not importing changes that have happened to the MAINline version of the default plugins.

The alternative, is to remove all plugins from the core release, adding them using their own set of zips and installers - which would help in the packaging too - but that then suggests to me that we should use a branching system more like debian.

-- SvenDowideit - 28 Sep 2007

Let's think a bit more about Sven's suggestion remove all plugins from the core release. This suggests that we are heading for a MANIFEST for the release build that contains only include lines. Thus in build terms, the core becomes almost identical to an extension. Right now, today, this could be achieved by moving the core code into - for example - twikiplugins/Core. The existing pseudo-install.pl can then be used to install the core. Then I can define twikiplugins/TWikiForGeneralUse, which is empty except for a MANIFEST which specifies the other modules to include to create the specific TWikiFor. The main thing that is missing is the ability to branch/tag just that subset of the repository that comprises the TWikiFor - and that can be relatively easily scripted, I suspect.

OK, so we have most of the tech to do most of this already, back to repository organisation. Right now we use the twikiplugins area as a catch-all for "everything that is not the core". This gives us the problem that when we pull in twikiplugins in order to get the default extensions, we also pull in a huge amount of garbage. So how can we organise the repository more flexibly? One possibility:

http://svn.twiki.org/svn/twiki/trunk/
   checkout.pl
   tag.pl
   Core/
   TWikiForMostPeople/
   CommentPlugin/
   TWikiUserMappingContrib/
   ...   

Now, to check this out:

$ cd devarea
$ svn co http://svn.twiki.org/svn/twiki/trunk/checkout.pl # bootstrap the checkout script
$ perl checkout.pl TWikiForMostPeople

perl checkout.pl TWikiForMostPeople can be quite simple. Pseudo-code:

svn co http://svn.twiki.org/svn/twiki/trunk/TWikiForMostPeople TWikiForMostPeople
for each 'module' !included in the MANIFEST for TWikiForMostPeople do
   # Obviously 'Core' is one of the modules in the MANIFEST
   svn co 'module' to  TWikiForMostPeople/twikiplugins/'module'
   pseudo-install the module
done

Now, if I want to add additional plugins to this dev checkout, I simply:

$ cd devarea
$ svn co http://svn.twiki.org/svn/twiki/modules/BathPlugin TWikiForMostPeople/twikiplugins
$ cd TWikiForMostPeople && perl pseudo-install.pl BathPlugin

So, that makes life relatively easy for the developer, for relatively little work. But what about the releaser? They want to be able to tag everything mentioned in the MANIFEST for the TWikiFor in one command. I doubt that svn copy is able to traverse between URLs, so again a script is required, but again it can be quite simple:

$ cd devarea
$ svn co http://svn.twiki.org/svn/twiki/tag.pl # bootstrap the tag script
$ perl tag.pl TWikiForMostPeople

Pseudo-code:

make sure the checkout is unpolluted
for each 'module' !included in the MANIFEST for TWikiForMostPeople do
   svn copy 'module' from the working area to the tag
done

Since the tag is a copy of TWikiForMostPeople, the Core, and all the default extensions listed in the TWikiForMostPeople MANIFEST, it is now ready for perl build.pl release.

Of course I realise there are niceties to observe - such as all modules must observe strictly non-intersecting sets of build files - but I'm sure those can be resolved.

-- CrawfordCurrie - 29 Sep 2007

Another approach might be to extend MANIFEST in the TWikiFor to include an SVN revision number. My gut tells me this is a bad idea, but maybe someone knows better.

-- CrawfordCurrie - 29 Sep 2007

Please let us not make things so complicated that only 3-4 people can ever understand what is going on. Thing can get overengineered.

The basic requirement for me is

• The core and default plugins is one entity that can be isolated in a patch branch.
• The non-default plugins should be in their own branch so that we do not have to copy/update/checkout that whole chunk every time.

It is very important that the default plugins are treated 100% like the core code so that the release branches can be stable branches while more risky development can occur in MAIN.

TWikiFor is a thing people like to talk about but noone really does. I cannot imagine many plugins in the default distribution anyone would want to live without. So TWikiFor is in practical a matter of adding plugins. Distributions like VMware distros are easy to make by installing TWiki and then installing the extra plugins. If you want to remove a plugin - just deactivate it - no need to uninstall.

Let us keep things simple. But I agree to split out the non-default plugins in its own branch. It also makes it simple to separate people with plugin only access and those with access to core (= everything).

-- KennethLavrsen - 29 Sep 2007

TWikiFor is a thing people like to talk about but noone really does. I cannot imagine many plugins in the default distribution anyone would want to live without. Huh? I have very few installations where ClassicSkin is required. Or EditTablePlugin. Or NativeSearchContrib. Or PreferencesPlugin. Or RenderListPlugin. Or SlideShowPlugin. Or TipsContrib. On the other hand I often want LdapContrib. And ActionTrackerPlugin. And EditRowPlugin. And other more specific, market-focused extensions.

Yes, I can disable the default plugins (though not the default contribs). But why am I forced to take all this unwanted code into my installs? The whole point of TWikiFor is to make it easier for people to configure TWiki for target markets and applications. What I am hearing here is "TWikiFor isn't needed because Kenneth doesn't need it".

-- CrawfordCurrie - 29 Sep 2007

I cannot see why it is so important that you can remove the code for a few extensions from the distribution zip. I can better see the point in adding a few. And today all I have to do is add ONE LINE for each additional plugin to the MANIFEST before I do the build. Pretty easy.

I am saying that TWikiFor is not very much needed because people have talked about it for years but noone has implemented it even though noone ever prevented them from doing it. And it was not one of the priorities in Rome either.

I am giving my opinion in the context of refactoring Subversion. I would not want to end up with an over-engineered solution that only a few people can figure out. All I am asking is for is that a refactoring is still kept simple.

And I would like it to be very clear which plugins are default plugins that MUST WORK and are treated with same care as the core code - and which plugins are just extensions that live their own life cycles.

-- KennethLavrsen - 29 Sep 2007

I can think two simple solutions:

Separate directories for twikiplugins and coreplugins.

The repository will look like this:

twiki
   | 
   | - MAIN
   |    | - bin
   |    | - lib
   |    | - etc,etc,etc
   |    | pseudo-install.pl
   |    | co-plugins.pl
   |    | - coreplugins
   |    |     | - CommentPlugin
   |    |     | - TablePlugin
   |    |     | - EditTablePlugin
   |    |     | - etc,etc,etc,etc
   |
   | - twikiplugins
   |   |  - AccessStatsPlugin
   |   |  - AddDBMGroupPlugin
   |   |  - AdvertsPlugin
   |   |  -  etc,etc,etc,etc,etc  

co-plugins.pl will perform a checkout of "twikiplugins" in the proper place, so pseudo-install.pl can be used. Also, a small modification to pseudo-install.pl needs to be done so it looks first in coreplugins and then in twikiplugins to install an extension.

Let's see how it stands against the requirements:

1. The set of people permitted to check into the extensions area must be different to the set of people able to check into the core
2. A checkout of the core must not include non-core extensions
3. A core checkout may include default plugins (core checkout WILL include default plugins)
4. / Individual extensions may be independently branchable/taggable Core extensions are branched/tagged with the core
5. It must be possible to run a TWiki directly based on a SVN checkout, i.e. without going through an build/install cycle first (perl scripts may be used, but must be portable to Windows)
6. It must be possible to run unit tests on a SVN checkout.
7. It must be simple to branch and tag a TWiki version with the 'default' Extensions included in the branch/tag,

Result: 5.5/7

Keep things "as is" and provide a script for the release manager to branch/tag a TWiki Release This script was already described by Crawford in his analysis/proposal (the tag.pl script).

Let's see how it stands against the requirements:

1. The set of people permitted to check into the extensions area must be different to the set of people able to check into the core
2. / A checkout of the core must not include non-core extensions This will be true for a checkout of the release branches
3. A core checkout may include default plugins (release branch checkouts WILL include default plugins)
4. Individual extensions may be independently branchable/taggable Core extensions are branched/tagged with the core
5. It must be possible to run a TWiki directly based on a SVN checkout, i.e. without going through an build/install cycle first (perl scripts may be used, but must be portable to Windows)
6. It must be possible to run unit tests on a SVN checkout.
7. It must be simple to branch and tag a TWiki version with the 'default' Extensions included in the branch/tag,

Result: 4.5/7

-- RafaelAlvarez - 29 Sep 2007

Btw, Crawford's proposal scores 7/7.

-- RafaelAlvarez - 29 Sep 2007

So what is Crawfords proposal? I do not understand it. Which is why I ask for things to be kept simple. If "remove all plugins from the core release" is part of it then I throw a core veto because this is will mean that noone will care anymore if the default plugins all work in a release.

-- KennethLavrsen - 29 Sep 2007

Sorry, I tried to provide an illustration of how my proposal would work, and that has ended up clouding the issue for Kenneth. The proposal is:

http://svn.twiki.org/svn/twiki/trunk/
   |
   +-- checkout.pl
   +-- tag.pl
   +-- build.pl
   +-- Core/
   |     +-- lib
   |     +-- data
   |     ...
   +-- TWikiForMostPeople/
   |         +-- lib/MANIFEST
   +-- CommentPlugin/
   +-- TWikiUserMappingContrib/
   ...   

Thus the MANIFEST for the main TWiki release is in TWikiForMostPeople. Since TWikiForMostPeople defines what you, Kenneth, call "the core", then nothing has been removed from "the core". In fact the MANIFEST is much simpler; all it contains is !include lines, which specify which other of the top-level modules to include. From a maintenance perspective, it's actually a lot simpler than what we have at the moment.

-- CrawfordCurrie - 30 Sep 2007

This leaves the problem with the default plugins being treated like core code in the release branch. Like the core code - it is important that in patch releases we can apply bug fixes to a default plugin, while making major refactorings and rewrites in the "MAIN" version of same plugin. We have done this in the past. You cannot just freeze on a certain SVN rev of the plugin. You have to be able to apply a small bugfix or security update to a default plugin in the release branch and at the same time the plugin may not even be working in the "MAIN" version because someone is in the process of rewriting it.

-- KennethLavrsen - 30 Sep 2007

Crawfords proposal can deal with release tagging and branching. The tag.pl script would create the tag/branch for the release by including in all the modules (core, and plugins) into the tag. Similarly, I'd add a build.pl, so that for us ReleaesManagers, we could:

• create a new release branch
  • ./tag.pl TWikiForSomething TWikiRelease04x03 (done in the trunk)
• build a test release
  • ./build.pl TWikiRelease04x03x02 (done in the TWikiRelease04x03 branch)
• create a new release tag
  • ./tag.pl TWikiRelease04x03x02 (done in the TWikiRelease04x03 branch)

also, another benifit of the create.pl is that it removed the 'work out howto use svn, then forget to run pseudo-install= problems that the less regular developers have.

-- SvenDowideit - 01 Oct 2007

It all sounds good.

But you still need to explain to me - where are the default plugins placed in the TWikiRelease0Xx0X branch?

How do you checkin small fixes to the default plugins in the release branch. I would like to see the release branch structure illustrated the same way as the trunk is illustrated above?

Will the directory structure of the default plugins be the same in "MAIN" (trunk or whatever) and the release branch? I would prefer - for simplicity that they are the same.

-- KennethLavrsen - 01 Oct 2007

IMO, I want release branches and tags to match whats in the zip, so that testing and working from it is as trivial as possible. This is the case for release tags, as I update them using the topic.txt,v files in the release tarball.

That said, I'm the most capable of working in whatever way the less knowledgeable prefer.

-- SvenDowideit - 01 Oct 2007

I want release branches and tags to match whats in the zip - no, that's just not good enough. You have to be able to reproduce the entire build process on the branch, and not just fight with the derived objects. A tag should just be a snapshot of a branch.

Kenneth, the release branch structure would be identical to what I described above, with the exceptions that (1) it would be a directory level lower, (2) it would only contain the released extensions and (3) it would not contain any TWikiFors. For example,

http://svn.twiki.org/svn/twiki/branches/
   +-- TWikiRelease04x03
   |   +-- tag.pl
   |   +-- build.pl
   |   +-- Core/
   |   |     +-- lib
   |   |        +-- MANIFEST (from TWikiForMostPeople)
   |   |     +-- data
   |   |     ...
   |   +-- CommentPlugin/
   |   +-- TWikiUsersMappingContrib/
   |   ... (but not including unreleased extensions)
   +-- TWikiRelease04x04
   |   ...

As Sven describes above, the release mechanics are actually two-stage, reflecting the process more closely:

1. perl branch.pl TWikiRelease04x03 branches a new release called TWikiRelease04x03
2. perl build.pl builds the release zips etc
3. perl tag.pl TWikiRelease04x03x00 tags the actual release

-- CrawfordCurrie - 01 Oct 2007

d'oh, good point wrt derived vs source objs. means that release tags need to be paired with some kind of derived set - so that builds can continue to add to the topic ,v's. no matter though thats a tiny detail I'll have to deal with off the side, perhaps in /svn/twiki/releases.

-- SvenDowideit - 01 Oct 2007

This proposal and discussion has been re-recreated in SingleBranchPluginDevelopment, and is scheduled for implementation in Feburary 2008.

-- SvenDowideit - 06 Feb 2008

This Plan has now been executed - now to update all our Documentation.

-- SvenDowideit - 17 Feb 2008