create new tag
, view all tags

Harmonize Newly Introduced Section Parameters With Existing Plugins

Reading the VarSTARTSECTION documentation, it appears that these variables serve the precise same purpose that the section commands in MultiEditPlugin serve. Therefore, it would be duplicating functionality and these two should be merged, which means that one or the other syntax should change. There is also some similarity to the section commands used by the SectionalEditPlugin but that one is focused on sectioning purely on outline levels.

This overlap should be discussed and resolved. Overlap and inconsistency between proposed and existing functionality should be eliminated.

-- Contributors: ThomasWeigert - 19 Oct 2006


It would be very harmful to change the syntax for SECTION feature that is in the core.

I have personally plenty of topics using this already. When this feature was introduced in TWiki4.0 we must assume that it is in general use by all installations since this is a core feature and not a plugin.

The SECTION tag follow the general syntax for TWikiVariables. It is used to include section of topics with INCLUDE and used heavily on templates in TWiki applications. %SOMEVARIABLE{"value" param1="value1" param2="value2"}%

There is a good API to decode this format and it is good to keep a consistant user interface.

So if someone wants to harmonize then it is should be the old plugins that use all sorts of syntax.

Then I hope the users of your plugins will be given a backward compatibility function so they do not have to replace tags in 1000s of topics. Should not be so hard to do.

I think it could be more interesting to not focus too much on this and instead focus on how to edit sections using an AJAX interface and have this added to core. I think there is more to win to coordinate that effort.

-- KennethLavrsen - 19 Oct 2006


  • I was not even aware of this feature and I don't recall any discussion when going to 4.0. Can you point to where this was added to TWiki 4?
  • The plugins were available long before TWiki 4, so when a tag was introduced then, if in fact it was, attention should have been paid to existing functionality. It is somewhat upsetting that identical functionality was introduced in Core without considering legacy. That is not in the spirit of the TWiki mission.
  • I don't like the idea of having two support two syntaxes in the plugins, as this means a slowing down performance by scanning for double the tags in different ways in an already inefficient handler. The right thing is to change this in one place or the other and provide a script to migrate topics.

This really was a poor process....

P.S. Even Sven seemed to be aware of the section feature, as he is also working on a plugin that overlaps this functionality...

-- ThomasWeigert - 19 Oct 2006

Some background moved from AddSectionParam...

This feature was discused in two topics: NamedIncludeSections and MultiStartStopIncludes

This overlap should be discussed and resolved.

  • The base intent of both VarSTARTSECTION and MultiEditPlugin are the same: The ability to define sections inside a topic. But there are differences on the final purpose. MultiEditPlugin and SectionalEditPlugin define the sections to be able to edit them individually. VarSTARTSECTION defines the section to be able to VarINCLUDE it in another topic. Neither of the plugin have a sintax to define the name of the section, and the named include is better done at the Core than in a plugin anyway. That is the main reasons of why I didn't extend the plugins instead of modifiying the core. The other reason is that at the time, those plugins where not working in Dakar. I don't think that there will be too much trouble modifying the plugin to support both the <section> syntax and the VarSTARTSECTION syntax, as they are both "equivalent".

-- RafaelAlvarez

And, I'd like to add that it also overlaps with the ajax functionality in InlineEditPlugin, and in the work we're attempting to do in the TopicObjectModel work. At best, I would suggest that there is zero need to talk of adding this to the core, as there are at least 3 bodies of work in Plugins, and moreso, that the work that both Thomas and I have done, is a superset of the proposal, and our work is a subset of what we are actually trying to achieve.

  • What does "zero need to talk" mean? Accept or discard a url parameter? -- ArthurClemens - 19 Oct 2006

-- SvenDowideit - 19 Oct 2006

by zero need to talk of adding this to the core, i'm saying that there is no need, there is ample work showing that it can be done successfully and well in a plugin. especially as the functionality you talk of should be done better using TOM - so I'd consider the work that Thomas and I have done as investigation that leads to the real solution. (Otherwise we have to support a potentially inconvenient code set in the core)

Also, the core principle that is being used on both inline edit and TOM involves no modifications or markers in the topic text. the more non-user info that gets added into topic text, the more difficult everything (user useage, our coding, our docco and maintainence) gets.

-- SvenDowideit - 20 Oct 2006

Copied from AddSectionParam by KennethLavrsen

To clarify, the overlap is not with the suggestion that Arthur had, but with the section feature which was introduced with complete disregard for existing plugins and now requires massive change of existing topics. An utter violation of the TWiki mission, I am sorry to add...

-- ThomasWeigert - 20 Oct 2006

First. How exactly is the INCLUDE/SECTION feature (which you just discovered after one year) suddenly going to break existing topics and why would your users have to make massive changes? Your plugins still work, right?

YOU are creating a problem here which is not there. It is your personal claim that the two features overlap and have to be merged.

  • He checked it in with SVN 6434. All developers should be subscribed to this mailing list. Again no objections. The code was checked in with documentation, test cases, the works. And this happened 19 Sep 2006. And again no objections.
  • The code was there all the way through beta releases in December and January. No objections.
  • And it was there when Twiki 4.0.0 was released very well explained in the TWikiReleaseNote04x00x00. At this point of time most of your plugins still had not been updated to work with TWiki4.
  • It is not the first time that cool details from plugins are merged into core. Why would this suddenly be a problem?
  • Your tags do not look like TWikiVariables. You use a html or xml like syntax. There is a concensus in the core development community to make the syntax for TWikiVariables consistant so the users have a chance to learn it.
  • IF the core developers had picked the exact same syntax as your plugins - ie. <section> then that would have interferred with your plugins so they no longer worked.

The section feature first of all takes in the function from MacrosPlugin which we actually still have installed because we have still have old topics that used it. And it still works.

But the design choice was wise because the include feature should not have two different tags. There is one: INCLUDE and it can include sections as well as entire topics. And on top of that it can include named sections and use parameters that become TWikiVariables in the included topic.

Let us say I have a hotline topic - very simplified - I actually have something like this in use where I include different content from different projects.

---+ Kenneth's Hotline

%INCLUDE{"ProjectA" section="schedule" }%
%INCLUDE{"ProjectA" section="summary"}%
%INCLUDE{"ProjectA" section="detailed"}%
%INCLUDE{"ProjectB" section="summary"}%
%INCLUDE{"ProjectC" section="summary"}%
%INCLUDE{"ProjectD" section="summary"}%

and the 4 project topics can look like this

---++ Project A - Schedule

| Milestone 1 | 10 May 1965 | | 12 May 1965 |
| Milestone 2 | 12 Oct 2007 | 10 Oct 2007 | |
---+ Project A - Status

Thing are going well

---+ Detailed Status

All the blabla that people love to write

---+ Issues

We are actually getting more delayed.

Nothing prevents you from adding <section> tags to the example above and you would probably not place them exactly the same places as the INCLUDE related START/ENDSECTION tags. And when you start using parameters with the includes then this becomes really powerful.

Thomas. I think you are jumping to conclusions here and I suspect that you really haven't spent time finding out what the new INCLUDE/SECTION feature actually does. I hope my little demo has convinced you that things are not so bad.

-- KennethLavrsen - 20 Oct 2006

Ken, I did look at this after this new feature became apparent. You are of course right that theoretically I would not have to change an could keep the old syntax for the MultiEditPlugin. But that would be stupid. Your example above clearly demonstrates this. The newly introduced section tag can be used not just for your includes, but also as the tag for that plugin, and it makes more sense to have a single mechanism rather than two as (i) there is less for the user to learn and (ii) the same section can be used in two ways...

I will go an change the plugin and have a script to change all our topics which are a lot. But I think that when we introduce new features we need think broader and look at the totality of TWiki not just our little idea....

-- ThomasWeigert - 20 Oct 2006

Originally I though about reusing the <section> syntax but chose against it (after a short discussion on TWikiIRC) because:

  • It's not possible to reuse the infrastructure for registering tags into the parser, that is way a lot faster than regex-in the topic several times.
  • The produced HTML will have an "unrecognized" tag in it, unless the tag is removed. If the tag is removed, the Plugin will just don't work. Unless the tag is removed at the end of the rendering(having yet another special case and yet another regexing). This unrecognized tag may be a headache for WYSIWYG editors and such.

Question is, for how long should we think? The older topic is from 2003, the implementation was 6 months old before TWiki04 went public.. and the syntax changed twice in that period (from SECTION to something to STARTSECTION) and I remember that there was a lot of discussion on that matter (how it should be called, how it should work, should it be a generic thing or a specific thing), but that discussion happened, sadly, on TWikiIRC.

One example of why we should return all discussion to Codev.

-- RafaelAlvarez - 20 Oct 2006

During all the Dakar discussion about named sections ( %STARTSECTION{...}% style) it never occurred to me that it could be used to mark up sections for editing. And apparently everyone else missed the potential overlap as well. All I had in mind, similar to Kenneth's example, was the possibility to selectively view parts of a topic - either by including them explicitly, or by using things like %INCLUDE{$topic section="glossaryentry"}% in a formatted search.

I recall that there have been discussions last year whether new TWiki markup should be XML/HTML like or TWiki like, and I seem to recall that there were pretty good points in favour of the %...% syntax, in addition to the rendering code in TWiki. Or at least, I decided to drop any ideas of using XMLish tags for extensions, even if they looked so familiar from Docbook or other dialects.

It would have been (and probably still is) rather easy to extend the %STARTSECTION% tag with an attribute edit in a compatible way. Maybe we need a way to "stack" tag handlers in the core, similar to calling a SUPER:: method in Perl. But this may allow to use TWiki style %% sectioning labels for SectionalEditPlugin in addition to the XML style - which I would consider much better than to teach TWiki core to render XML style tags.

As a final rant I could point out that this is another example of why PluggableIsHarmful without PluggableDocumentationArchitecture. One can not expect core developers to know about all variables in all plugins, so a comprehensive reference of all available tags would be very welcome.

-- HaraldJoerg - 20 Oct 2006

I don't think anything special is required for being able to edit. I'd say that the plugin would trigger off the section if it is on. Personally, I don't understand why there is even the different commands "include", etc., on the section.

On the syntax, I would normally not care, wouldn't it be for backwards compatability. For some reason, there seems to be a pattern that people have been using xml style syntax when there was opening and closing brackets, and TWiki style syntax otherwise. In other words, typically TWiki tags tend not to extend beyond a single line. It would not hurt to have more uniformity.

-- ThomasWeigert - 21 Oct 2006

TWiki tags that has parameters (like %INCLUDE% and %SEARCH%) already have multiline support. You can write something like this:


-- RafaelAlvarez - 21 Oct 2006

Yes, of course. What I meant was begin/end tags. start/stop include is the exception. Note your use of <verbatim>/</verbatim> in your example....

-- ThomasWeigert - 21 Oct 2006

Now I see what you mean. And you're right: The only TWiki tags have "content" are the STARTSECTION/STOPSECTION and STARTINCLUDE/STOPINCLUDE tags.

-- RafaelAlvarez - 22 Oct 2006

The STARTSECTION/STOPSECTION as well as STARTINCLUDE and STOPINCLUDE have been part of the core feature in released versions of TWiki. Same with <verbatim>/</verbatim>.

We have to pay respect to the 1000s of installed TWikis and their 10000s of users. And we cannot survive if we start changing things this basic in a non-compatible way. Remember - to the users the real values lies in the many many hours spent writing topic contents. Not TWiki itself.

It has nothing to do with what is right or wrong or what we like today. We cannot rename <verbatim>/</verbatim> to %STARTVERBATIM% and %ENDVERBATIM% for the same reason.

"Protect corporate investment (topic contents) from data corruption and incompatible changes" is a matter of life or death for a project like TWiki.

This is why it is important that we always think carefully about things and review proposals well before we include anything in the core code or in the default plugins.

For the non-default plugins it is up to the plugin authors to follow the TWiki Mission in every aspect. But Thomas, you should know that your plugins are probably widely used by many TWiki installations and their users. Each time you make a non-compatible change, you cause a lot of pain to the end users.

It is not very friendly to demand that the admin has to run all sorts of more or less well working upgrade scripts, each time he upgrades either TWiki or a plugin. If a conversion is needed it must happen on the fly during normal like the new topic format 1.1 in TWiki4 is a good example of (spaces instead of TABs).

I am not sure where this topic is heading, but let me make it clear that any attempt on changing basic core Twiki syntax which is assumed to be broadly used - will be very anti-mission and most like met with more than one veto from the core team members.

But do not let this stop the discussion because the generic part of this discussion is important for both plugins and for future features added to core, and it is especially important to understand the pros and cons for the two different formats. And maybe there is still a good reason to use both in different situations.

-- KennethLavrsen - 23 Oct 2006

Understood, but here is the flip side:

  1. Users need to needlessly learn multiple commands when one section command would do. (Again, whoever introduced the %STARTSECTION% was careless; but what is done is done.)
  2. The already slow code is slowed down even more when we have duplicate commands for backwards compatability.

-- ThomasWeigert - 23 Oct 2006

Sidenote: TWiki is a community driven project, it is not always possible to get ones ideas accepted. For example, I would have preferred to call the section identifiers STARTSECTION and STOPSECTION (instead of ENDSECTION) to make it consistent with the existing STARTINLUDE/STOPINCLUDE variables. Now users need to remember that there is an exception: Some variables have a STOP* prefix, others an END* prefix. It is possible to remember this, but then, one needs to remember it, or look it up every time.

I am with compatibility. We decided to release STARTSECTION and ENDSECTION with TWiki 4.0, we cannot replace it with something else anymore.

Lessons learned with this story:

  1. Get involved,
  2. Look at many alternatives,
  3. Design carefully,
  4. Debate,
  5. Decide,
  6. Release,
  7. Stick with it.

-- PeterThoeny - 27 Oct 2006

To be honest, I'd rather write a script that changes all my topics that use ENDSECTION to STOPSECTION than remember that this command is different.

Consistency in the language is extremely important, and a big obstacle for users if not present....

Here is a sample:

  1. Use of Twiki tags vs. HTML like tags (STARTSECTION vs <section>)
  2. Use of inconsistent keywords (STOPINCLUDE vs ENDSECTION)
  3. Inconsistent use of attributes (row x col in TWikiForms, col x row in EditTablePlugin)

Over and over my users triip over these kind of things, or at a minimum, waste time having to go to find the manual....

If we can fix things by cleaning up an existing database by running a script over it, we should do. I have done this several times on TWikiApplications we built, and the benefit for the users is huge...

Vendors do the same thing... they introduce new formats where sometimes data conversion is required... in the long run users benefit....

-- ThomasWeigert - 27 Oct 2006

I am for this proposal.

-- ArthurClemens - 27 Oct 2006

Funny. Noone mentioned the inconsistant STOP vs END before Peter brings it up. No users every complained about it. And now it is suddenly a huge problem and the TWikiMission is for sale again.

So now we have a proposal that requires that users have to re-learn what they already learned the past 8 months.

The admin has yet another obstacle to upgrading. They do not even apply the hotfixes even though I have tried to make upgrading easier. TWiki is still a geeky piece of work to install and upgrade. Just stick around #twiki on IRC. No they are not idiots all of them!

And it is almost impossible to make a script that gets everything. For example comment plugin templates where people put in a in the middle of the tag to prevent expansion - even when not needed -and we do not know where. Or formatted searches where people put in $percntSTARTSECTION. Something always goes wrong with these scripts. There is always something the scripts miss and then afterwards the admins that are not programmers and not familiar with all the secrets and features of TWiki end up with trouble. It is not easy to run a script. From which directory do you run it? Do you have the CPAN libs needed? Does the INC path find what is required? Does it work on a Mac? Does it work in Windows?

And when we hack the topic, what about the history? Do we update the history? Do the users want all their topics suddenly upreved all on the same day? I would not want it. OK then we just hack the latest version but then why has the section tags changed since previous version? And why does copying the version two times back not work? Why does picking a page from a non-upgrades TWiki not work in our TWiki? Just run a script. It sounds so easy. It is not. For TWiki developers sitting with the noses deep in the code every day any glitch or a quick hack is not an issue. But to normal admins and to end users the important thing is stability.

This section example is not the only inconsistancy in TWiki. If we start on this we can soon replace 30% of the TWiki syntax. Example verbatim also.

... Decide, Release, Stick with it. That is also my oppinion.

-- KennethLavrsen - 27 Oct 2006

Noone is arguing the mission. But we can deprecate things. Let new STOPSECTION work exactly the same as old ENDSECTION, and update the documentation. Same goes for other inconsistencies.

-- ArthurClemens - 27 Oct 2006

I tried to make it consistent by suggesting to name the variables STARTSECTION and STOPSECTION, but I was overruled. The decision has been made, lets stick with it. And going forward, lets design enhancements more carefully.

Using upgrade scripts to fix topic content because of spec changes is out of question, there are too many headakes for existing admins/users (looking at older topic versions, consolidating TWikis, sharing content across TWikis, company mergers, etc.).

I would find it OK however to introduce a STOPSECTION and to keep ENDSECTION as an undocumented variable until, say, 31 Dec 2099 smile

-- PeterThoeny - 28 Oct 2006

From my experience, deprecation never makes anything vanish, and upgrade scripts can't work sufficiently reliably (wanna globally change ENDSECTION to STOPSECTION? Bad, bad idea because you're going to make the topics documenting the change pretty incomprehensible). I support Peter's suggestion: If we want STOPSECTION, keep 'em both. But I'd even go further document ENDSECTION, if only to say "This variable should be replaced by =STOPSECTION=". But alas - I could always add a VarENDSECTION topic to my installation if you agree that it should be undocumented.

-- HaraldJoerg - 28 Oct 2006

Is having TWO different TWikiVariables that mean the same thing going to really help anyone? Isn't it going to be more confusing instead? I have my doubts but would not vote against such a proposal.

This discussion topic has drifted away far from its original starting point and I think we have lost many on the way. I would prefer closing the reference to this topic on WhatIsIn04x01 as rejected and if anyone wants the TWikiVariable STOPSECTION added being an alias for ENDSECTION then please raise a topic specific for that on WhatIsIn04x01 so we can have the entire community decide explicitly on that. It will be an easy proposal to understand and vote on and I doubt any core team member will veto it. But remember that if you raise such a proposal on WhatIsIn04x01 you must commit to implement it in near future for 4.1.

-- KennethLavrsen - 28 Oct 2006

I think the lesson to be learned from this is that an important part of the design of a system like TWiki is also the design of the language that users are confronted with. We need to make the design principles of the TML a part of the specification, to help both developers and plugin developers to pick the right keywords. At the point when the decision was made to call this tag ENDSECTION it would have been easy to check against the principles and realize that one should go with STOPSECTION. Or when a plugin developer decides to use <section> (as I did) because he or she is under the impression that the accepted pattern is such, a written down design principle would help make or avoid that decision.

-- ThomasWeigert - 29 Oct 2006

See http://develop.twiki.org/~twiki4/cgi-bin/view/Bugs/Item1484 for the discussion on these variables. Thomas, you are dead right about the need for some consistent rules on syntax. However, the legacy is such that TWiki will break the rules even before they are written. Consistency is not a strong point of TML.

-- CrawfordCurrie - 29 Oct 2006

While it may have wandered away from the original sentiment: this discussion has uncovered many gems along the way. It would be worthwhile to refractor this page and somehow put those gems into DocumentMode for the development team / process.

Instead of losing me along the way, I've consistently looked forward with anticipation to what might be the next turn in this conversation. As a twiki admin and an end user of the SectionalEditPlugin, I have many times been tempted to comment here. But at each juncture I was torn about which side to speak out in favor of. And it is exactly because both sides are valid that the best way to harmonize the section parameters is to make them both valid. The only argument against BOTH might possibly come from the performance camp, but even still. When in doubt, BOTH can be a great answer to inconsistency.

As a user, I don't want to have to go to the help pages at all. And if it has been a coupla months since the last time I used a section .. all I want is for the thing to work. So whatever variable I guess at using first, be it STOPSECTION, or ENDSECTION ... I just want for it to work so that I can keep moving without looking up the answer to why it doesn't work.

-- KeithHelfrich - 29 Oct 2006

Keith, thanks. I will code up SectionalEditPlugin and friends with both syntaxes and run some performance tests to see the difference it would make to offer both variants (not immediately as I don't have much spare time right now)...

-- ThomasWeigert - 29 Oct 2006

Good move Thomas.

I have made a Codev topic BlocksAndInconsistentTML which is a helper topic for both this topic and others and misc bug reports.

It also shows that it makes little sense to make an alias STOPSECTION that means the same as ENDSECTION because this is just one of many similar inconsistencies. I spent many hours gathering the data because I wanted the overview myself. And now I am convinced that

  • ... Decide, Release, Stick with it. as Peter said is the right thing to do
  • ... Consistency is not a strong point of TML as Crawford said is a thing we will have to live with when it comes to the past. We can try and make it better in the future with future features.

Take a look at BlocksAndInconsistentTML, and I hope that this long discussion can rest now. We have a lot to focus on to get 4.1 released before new year.

-- KennethLavrsen - 29 Oct 2006

TopicClassification BrainstormingIdea
TopicSummary Overlap and inconsistency between proposed and existing functionality should be eliminated.


Edit | Attach | Watch | Print version | History: r28 < r27 < r26 < r25 < r24 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r28 - 2006-10-29 - KennethLavrsen
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.