Introduction Motivation Previous Discussion Syntax Discussion of Syntax Use Cases Comments Implementation Defining Sections Example Usage Including Sections Using Named Sections as Search Results Latest Headlines Notes on Search Further Development Overlapping Sections INCLUDE section is a regex Implementation Notes Heading 1 Heading 2 Heading 3 How does it work Yada Yada Yada Discussion. (on a dev page How to use Remote Includes Category: TWikiPatches

Introduction

The patch attached to this page is against a fairly recent alpha. It's worth noting that it addresses the concerns that were raised about the original patch, has been in use for many months, and allows a number of seriously interesting uses. Overlapping sections has been found to be an essential requirement in practice when repurposing content.

The original patch needed chopping down somewhat - I've lopped off around 1700 lines, to what is probably a minimal patch. (Might've chopped out too much) 18, Dec 2003

This proposal is to extend TWiki's syntax slightly to make dealing with sections of topics simpler & easier. The idea is to name sections of a page for inclusion into other pages. Editting facilities for sections already exist as plugins. Support for jumping directly to sections (headings) also already exists. Treating sections as directly includable objects is the next logical step.

Motivation

Whilst performing selected inclusions using the pattern parameter works for this, and is therefore sufficient from a technical perspective, it is by no means a satisfactory solution for the majority of users. For pages outside a TWiki web where structure cannot be imposed, it's great. For local pages, allowing (but not imposing) structure is a good thing that can be taken advantage of.

Also, in order to use the pattern parameter it requires end users to understand the lexical structure of the page and regular expressions. Whilst I suspect a number of the viewers of this web understand the fact that:

A

pattern="^.*STARTSECTION=Example(.*)ENDSECTION=Example"

B

section="Example"

Would achieve the same result, I've generally found that the majority of contributors to a Wiki simply don't make use of the pattern paramter. (I've found it's useful for "wiki masters", but few others)

Also pattern A is more akin to <b> hello </b> rather than *hello* in terms of the complexity of what you're asking the users to type

Previous Discussion

As a result I would say that the pattern parameter is not sufficient from a twiki web contributor's perspective. Looking through the references from LynnwoodBrown, I see that I'm at least the third person to suggest having a way of denoting sections in TWiki for manipulation. There have been a number of suggestions as to how to go about this, and even some code in the area

The simplest suggestion is essentially the same as the one suggested above - use STARTINCLUDE to denote sections & extend the INCLUDE syntax.

A system that has been found to work by MikeBarton was discussed in the ConditionalRendering topic - essentially you take a topic that you would denote sections in, split it into a collection of topics along the lines of TopicNameSectionName, and then use INCLUDE & URLPARAM to pull in only the sections required. This does indeed split sections off, but is IMHO clunky. Also the conditional rendering plugin performs this task as well, but (again IMHO) in a cleaner way.

The third option is to use a new tag to denote sections. The has been suggested more than once in more than one way:

• The author of the conditional rendering plugin suggested the use of a %SECTION% tag.
• The TocPlugin also uses a section tag, but of the form %SECTIONN% along with 3 other tags.
• The standard %TOC% tag denotes implicit sections delimited by header tags.
• The SectionalEditPlugin defines an XML-like tag for denoting divisions between sections (ala <hr>), but limits the use of these tags for denoting sections that can be editted. Whilst these sections are edittable, they aren't includable.

Based on all this, I would suggest that rather than modify the STARTINCLUDE/etc tags, it might be simpler to denote sections explicitly - using the idea of a %SECTION% tag. (Preferably accompanied by an tag, rather than just act as a divider) The SectionalEditPlugin can be modified to accept the new syntax, and this would also start to nest nicely with the CommentPlugin.

Since TWiki already has sections in terms of automated anchors for %TOC%, I would suggest that modifying %INCLUDE% rather than creating a plugin to handle a new include tag along the lines of %SECTIONINCLUDE%. The reason for this is to eliminate duplication of code.

Syntax

Two kinds of pages:

1. Those which are divided into sections. Boundaries between sections are denoted by a %SECTION{name="foo"}% tag. With default names for sections being section1, section2 etc.
2. Those that contain sections. For these pages, not all text is part of a section. This scenario is equivalent to using STARTINCLUDE. For this syntax the beginning of a section is denoted using %SECTION% , and the end is denoted using %ENDSECTION%.

Syntax of %SECTION%

%SECTION{name="name" depth="level"}%

Syntax of %ENDSECTION%

%ENDSECTION%

Syntax addition to %INCLUDE%

%INCLUDE{"SourcePage" section="name"}%

Discussion of Syntax

Unlike EdgarBrown's suggestion this disallows overlapping or nesting by default. (At least in the first instance)

The name for include should be allowed (at some point) to extract sections denoted using TOC sections - ie heading tags - as well.

Use Cases

Generation of newspaper-like wiki sites:

• Pages divided into headline (and byline), firstpara & remaining sections.
• Front page of site pulls out headlines & first para sections. And formats nicely.

MattWilkie misinterpretted this as meaning multiple columns, and gave a nice CSS way of doing multiple columns - see TWikiUsingCSS.

Generation of user docs from feature documents:

• Features could be documented with sections: syntax, introduction, longdescription, discussionofimplementation, usecases, examples

• User docs could then just pull out the introduction, syntax, usecases and examples sections. You'll note that this actually matches this proposal document. If the syntax was accepted the implementation docs drop out quite well...

---

Comments

Well, you have my vote! I raised essentially the same idea last year in MultipleCustomIncludes.

-- LynnwoodBrown - 10 Jun 2003 [ This refers to the STARTINCLUDE{section.. } style syntax ]

The pattern=".." paramter in an INCLUDE can be used to extract an arbitrary part of a topic. Would that be sufficient? But then, named sections would be more convenient.

-- PeterThoeny - 10 Jun 2003

Essentially this is discussing of how to achieve FineGrainedAddressing in twiki. GrantBow has started the PurplePluginDev to explore the idea (still at the brainstorming stage; no code yet).

-- MattWilkie - 10 Jun 2003

I've just downloaded, installed and played with purplewiki, and I do think it's got some interesting aspects to it. As a test however, I took the source for the revisions of this topic (1.1, 1.2, 1.3, 1.4, 1.5, 1.6) and tried editting the purple wiki pages as you would a twiki page, including refactoring.

Some initial thoughts - for documents that don't change much, having highly granular references, and in discussions where refactoring doesn't happen much I think they're good thing. For documents that go beyond a few revisions though, you lose NIDs - ie URLs that worked no longer do. (At least not with PurpleWiki) The reason for this is the NIDs that were in use just drop off - content gets deleted. (Allowing NIDs to jump into revisions though would resolve that issue)

Also the IDs generated end up relating to either the structure or just the fine grained elements, rather than generic concepts - like "introduction", "executive summary", "abstract", "examples", and so on.

The concepts in purple numbers are this:

• NIDs - essentially refer to specific paragraphs - and hence can be viewed as a hash - and as a result should refer to any copy of the paragraph anywhere on the site. (Perhaps with one "original" location) Changing the position of a paragraph should not change it's NID.

  NIDs could be potentially calculated by transforming paragraph text to plain text, soundexing the text and then MD5 or MD2'ing the resulting text.

• HIDs - these again refer to specific paragraphs, but refer to structure. If you have 3 sections with 2 paragraphs in each, with a 4th section with 2 subsections each with 2 paragraphs, you would have HIDs of 1, 1A, 1B, 2, 2A, 2B, 3, 3A, 3B, 4, 4A, 4A1, 4A2, 4B, 4B1, 4B2

  In this situation if you switched text in paragraphs 2A and 4B1, the HIDs of the 2 paragraphs would stay the same. (A HID just says "take me to section 3, paragraph 1", whatever the content)

Both are useful in many ways, however both are very machine oriented. How do I know that section 3 of all documents is always going to be an example/usecase ? HIDs & NIDs are useful in their own right - but I feel are orthoganal to the idea of having named sections.

All said and done, I now want to be greedy - I want both section names and purple numbers - for different purposes I'll let GrantBow work on the latter though since he's been thinking about it longer.

-- MS - 10 Jun 2003

Implementation

I've implemented a first version of this, along with a patch to extractNameValuePair to allow %INCLUDE%'s to appear in %SEARCH% formats.

• To use this patch:
  cp NAMEDINCLUDES.patch _<twiki>_/lib
patch -p0 <../NAMEDINCLUDES.patch

This patch has been created againt BeijingRelease .

Defining Sections

%SECTION{_name_}%	Start a section called name
&ENDSECTION%	End the last defined section (section's can't overlap)

Example Usage

%SECTION{Headline}%

Test

%ENDSECTION%
%SECTION{Comment}% This is a page demonstrating named sections. %ENDSECTION%
This text will not be included.

This renders normally - with the extra TWikiVariables replaced with empty space.

Including Sections

Assuming the above example existed on a page named NamedIncludeTests.

You type:

%INCLUDE{"NamedIncludeTests" section="Headline"}%
%INCLUDE{"NamedIncludeTests" section="Comment"}%

You get:

Test

This is a page demonstrating named sections.

Using Named Sections as Search Results

The following search looks for topics with a Headline section, and pulls them in as search results, along with any comment and firstparagraph sections. Also the Headline is turned into a hyperlink to the topic.

_Latest Headlines_
<ul>
%SEARCH{ "SECTION{Headline}" limit="3" scope="text" \
order="modified" nosearch="on" regex="on" nototal="on" \
format="[[$topic][%INCLUDE{'$topic' section='Headline'}\%]] \
%INCLUDE{'$topic' section='Comment'}\% %INCLUDE{'$topic' \
section='firstparagraph'}\%"}%
</ul>

You get as a result (for example):

Latest Headlines

Random Test

This page demonstrates named sections. So this is a named firstparagraph for dealing with various bits and bobs... So there you go.

Random Test Too

This page also demonstrates named sections. So this is a named firstparagraph for dealing with various bits and bobs... So there you go.

Another Test

This is another page demonstrating named sections. I've now implemented named sections - it was rather trivial, but incredibly useful.

Notes on Search

If you use an INCLUDE inside, you need to escape the end of the %INCLUDE{}% with a backslash - ie %INCLUDE{}\% . If you don't do this the search expression (and hence your format) terminates early, doing Not What You Want. Also since quotes necessary for the INCLUDE cannot be escaped, you have to use a single quote ' where you would use double quotes ". These are converted to "s before they are needed. (See patch)

Further Development

Overlapping Sections

The current implementation doesn't allow overlapping sections, multiple sections starting & closing at the same points. I've knocked up a modification that allows the following which I'm currently testing:

---+ %SECTION{fullstory,title}% My Interesting Story %ENDSECTION{title}%
%TOC%

%SECTION{teaser}%This story is really interesting. It's so interesting
that if I really told you every thing here what the story was about then
your brain would just melt. I mean, if you think that Belgian Chocolate
is good, then this will blow your mind. %ENDSECTION{teaser}%
---++ Actual Content
OK, I lied, this story is actually worse than watching paint dry.
%ENDSECTION{fullstory}%

When combined with WebForms and the search aspects above you can actually end up with a fairly sophisticated but simple to use, extend, modify & loose content management system. Whilst the above example shows wellformed nesting in XML/HTML terms, this does NOT need to be the case. The following is perfectly legal:

%SECTION{one}%
%SECTION{two}%
%SECTION{three}% 
%ENDSECTION{one}%
%ENDSECTION{two}%
%ENDSECTION{three}%

The reason for this is twofold:

• It's easy to implement. (Key point )
• Actually allows people to do things nesting wouldn't. For example I might view sections 15.3,16.1,16.2,17.1 of a document as an important chunk of information, but the original author viewed them as non-overlapping pieces of prose. The advantage this gives is that multiple people can define multiple sections of interest to them for inclusion into other pages without interfering with others - even if they overlap.

INCLUDE section is a regex

This change has been made to simplify handle overlapping sections. One massive change this does is change the semantics of this: %INCLUDE{"topic" section="Example"}% from simply meaning include the section named "Example", to actually mean "include the first section from the topic with 'Example' in the section name". (It also means that actually using a regex for the section name is valid)

Implementation Notes

Not posted up here yet. Reason is the plans for changing the implementation:

• Take the key code out into plugins.
• Provide the necessary hooks for the plugins in the appropriate places in the core twiki code.
• Provide hooks for section handling. (This would allow DIV's, stylesheet classes sections to be specified for sections by plugins etc, as well as making edit sections more integrated)

-- MS - 02 Jul 2003

The code that is in use at http://www.yeoldeclue.com/ uses a radically different version - it allows:

• Multiple section start at the same point
• Named end sections
• Multiple section stop at the same point
• Overlapping sections
• Include sections parameterisable based on URLPARAM. (Try it with the existing setup - you'll find it breaks quite badly)
• Furthermore it also adds in plugin hooks so that people can hook extra processing into sections.
  • This allows the sectional edit stuff to integate cleanly
  • This allows folding sections to be done cleanly
  • This allows for systems like you get at various news sites where a page is viewable over multiple pages, or on just one, and also allows for a site at have parameterisable searches. (eg do you want headlines, summaries or full articles) Having plugin hooks also allows people to potentially specify an diff section instead allowing people to only see changes on all the pages instead.
    • NB I've not written these plugins, but they are made possible via these hooks. (I've done some work on folding though)

Since there's clear interest in this piece of code I'll upload the version that's in use shortly. (I can give read only rsync access to those interested to the code live on www.yeoldeclue.com) I'll update the docs (or someone else can) once I've done that.

The changes that allow this are fairly small, but allow a number of quite interesting things as indicated above.

Incidentally to skin writers - this also allows much, much, much more flexibility for you. (Though not as much speed at present as standard skinning approaches)

-- MS - 01 Aug 2003

IMO, supporting overlapping sections is overkill, if needed in rare cases it can be done with a pattern include. It also creates a problem if a future TWiki uses XML natively, and for XML converters, e.g. DocBook.

-- PeterThoeny - 03 Aug 2003

Overlapping sections is a pretty important requirement when all is said and done. The implementation is trivial - far simpler in fact than enforcing XML constraints. It also allows people to decorate an article. For example the following example would be from (say) a website that contains articles. (Playing around with similar ideas at www.yeoldeclue.com) This might be viewed as OTT, but it allows some interesting pages - I might even add some examples there at some point.

%SECTION{fullarticle,summary}%

%SECTION{articleheadline}% Title

Abstract

%SECTION{abstractbody}%In this document we discuss the following aspects of the system:
• It slice
• It dices
• It cooks too!

Some random text

Lots of interesting discussion.

Conclusion

%SECTION{conclusion,conclusionanddiscussion}%In this document we discussed:

• X, Y, Z
And found that this was highly beneficial.

Discussion

%SECTION{discussion}% Whilst XML is generally a good format for many things, for some things - such as greater semantic overlay - specifically overlapping sections it's a very poor fit.

-- TWikiGuest - 01 Apr 2004

That might seem like a non-ideal version - but in practice some content management systems for some professional systems generally require the person creating the content to enter information multiple times in multiple places rather than simply allowing overlapping sections.

Further examples of overlapping sections being useful:

%SECTION{shortversion,webhomeversion}%

Welcome

This is TWiki, a tool for collaboration. The basic ideas are:

• Everyone can and should contribute - every page can be editted in your browser
• It is smart - it turns things you type normally in email into stuff that looks pretty
• It knows what pages are stored and automatically
%SECTION{crib}%

Crib sheet

You type:

You type:	You type:
* This * That 1 Foo 2 Bar * Other ---+ Heading 1 ---++ Heading 2 ---+++ Heading 3	This That Foo Bar Other Heading 1 Heading 2 Heading 3

In this Web We Discuss Etiquette of Signatures

%SECTION{tease}%Sprawling an opinion all over someone else's website is the sincerest form of disregard for the thought "You are not the world", whilst you have a right to speech, people have a right not to listen. You're welcome to run your own tree. People don't agree on everything. They might do things slower than you like. If you don't like it don't use their tree or work with them.

This version can then be included in lots of WelcomeGuest topics with localisation of the content - in this case with the local version being the "tease" section(much like you can with a standard start/stop include). However, the very beginning can also be pulled in for webhome, and the cribsheet can be pulled into the edit template. (We do this and it makes user's live's significantly simpler - most people do not want to learn/memorise TWiki markup . And yes, this criticism applies to this feature at present as well - hence why I'd not uploaded the most recent code yet - TWiki variable syntax is an easy cop-out.)

Finally a more compelling argument is in the case of discussion revolving around a larger document. For example:

Slide Show Plugin

This is a plugin for doing slideshows

How does it work

The plugin looks for headings and bullet lists, and uses these to detect content and new slide beginnings.

%SECTION{contentiouscomment}%This is because %SECTION{context}% users are too stupid to use ideas like <STARTSLIDE> and <STOPSLIDE>. However, you could better argue that you have to consider the point that TWiki is designed to try and pick up contextual cues rather than have lots of cryptic comments everywhere.

Yada Yada Yada

Slide Show Plugin Dev

Discussion. (on a dev page

I've got to say, calling users stupid:

 

%INCLUDE{"SlideShowPlugin" section="contentiouscomment"}%

Is highly insulting thing to say. Are you all really that arrogant to think you're all that much more intelligent?

-- TWikiGuest - 01 Apr 2004

Hey - you must've missed the context - the comment continues:

 

%INCLUDE{"SlideShowPlugin" section="context"}%

It's not saying users are too stupid - it's saying that people don't want to think - not the same thing at all. That includes people who write code for their local TWiki installations.

In this example you can easily imagine the slideshow plugin page being initially created with no sections. Then the TWikiGuest gets uppity, puts markers around the section they find contentious and add their comment. Then the understanding person comes along, adds in extra section markers detailling the parts of the argument they find important and pulls that in.

Likewise when discussing documents people will often discuss things in terms along the lines of (say) sections 16.3 through 18.2. To the original document author that wasn't a discrete chunk. To a third party discussing (and hence using/manipulating/referencing) the document it is. This maps very poorly to XML. This is one way XML sucks for this sort of thing. (XML is good at many things, but not this)

In short - you can type the text once and repurpose it in lots of different ways - including well beyond those originally envisioned by the original author. Includes based on TWiki variables also allows simple, medium & advanced user interfaces, or even complete customisation far, far, beyond the current skinning approach. (After all easily pulling in easily user edittable, easily nameable content based on URLPARAMs and WebPreferences is a pretty powerful construct.) Such interfaces have varying degrees of help, and by definition require overlapping (and in some cases non-nested) sections.

The weather's been good here this weekend, hence the lack of upload.

-- MS - 03 Aug 2003

> Note that, unlike the attache patch, the \ backslash filter and ' to " filter is not appropriate
> because this would break existing content (a pattern parameter that has a \s* for example, or a ').

I'll revisit that part of the code then - backslash quoting is by far the most natural approach for such a thing. (Is it $percent, $prcnt, $percnt or $pcnt , or \% - like in the majority of progamming languages ? ) For the newcomer simply saying "put a backslash in front" pretty much simplifies things alot...

-- MS - 13 Aug 2003

This looks rather interesting, and appears to be an implementation of Google:transclusion, a concept invented by Ted Nelson who some believe was the inventor of hypertext/hypermedia - no time to comment now, though, as I should be at a leaving do!

-- RichardDonkin - 29 Aug 2003

> I should be at a leaving do!

Have fun

Transclusion's Interesting idea. (meatball wiki reference) TWiki has of course had this (with limited broken functionality :-/ ) since the remote includes were added. (Though most users can't do or undertand regexes of course so the functionality is limited normally for most users...)

I simply was thinking of the idea in terms of if you have a DOM of TWiki (hence the TWikiStructure topic), then named include sections (and fine grained addressing) are logical bits of the page in the same way the "foo" in X.foo is simply part of the object X.

The interesting thing is that remote transcludes are relatively easy to do using this as well - though it does require the remote site to provide a means of formatting the section. I've implemented this on http://www.yeoldeclue.com/ on the page http://www.yeoldeclue.com/twiki/bin/view/Yeoldeclue/AnySectionAnyTopic. The nice thing about this is that if (say) you wanted to include the synopsis of the CiteSeerMostCommonlyReferenced topic on yeoldeclue in a TWiki page, all you do is:

%INCLUDE{"http://www.yeoldeclue.com/twiki/bin/view/Yeoldeclue/AnySectionAnyTopic?requesttopic=CiteSeerMostCommonlyReferenced&section=synopsis&skin=plainnoextras"}%=

(contents of above include)

The URL's are a bit of a pain, but the creation of them can be automated using a simple UI - as I have done on HowToUseRemoteInclude - the user selects the URL for topic, section & skin, and hits "go". They can then copy the URL into their site.

Alternatively if they simply want to allow users to be able to select various topics they could simply include the UI on their site...

%INCLUDE{"http://www.yeoldeclue.com/twiki/bin/view/Yeoldeclue/AnySectionAnyTopic?requesttopic=HowToUseRemoteInclude&section=clueselect&skin=plainnoextras"}%

This would look like this: (given remote includes don't work on Twiki.org due to bugs and proxies...)

HowToUseRemoteInclude : clueselect How to use Remote Includes Select Topic... How to use Remote Includes Important resources and useful papers Interesting Interviews User Interface Wheels YAMI: Yet Another Messaging Infrastructure Online Programming/Technical Books Embedding a Python Interpreter in Python Apps Hypersonic Audio Processor Text Processing with Python Making Noises Select Section... Full Story "}% Synopsis "}% Select skin default "}% includeable "}%

More fantastically, you can combine this with URLPARAMs on the local server for selecting pages on the remote server for inclusion - after all there's no reason the remote UI can't have sections for each drop down. eg:

% INCLUDE{"http://www.yeoldeclue.com/twiki/bin/view/Yeoldeclue/AnySectionAnyTopic?requesttopic=%URLPARAM\{'requesttopic' default='HowToUseRemoteInclude'}\%&section=%URLPARAM\{'section' default='clueselect'}\%&skin=plainnoextras"}%

Named sections are a really simple little addition, but the effect on how you can use TWiki is quite, quite dramatic, as I hope the above description shows.

For example - imagine pulling in a search of the top stories from your favourite 5 newspaper sites into a single page, clipping that to a PDA/laptop or even printing it out, and reading on the train - all from a single personal site. The interesting thing as well is that the remote site retains control over presentation & look - which is something commerical sites would require for this kind of story syndication.

-- MS - 29 Aug 2003

Take it, leave it. Merry Christmas - I and many others find it useful.

-- MS - 18 Dec 2003

I like overlapping NamedIncludeSections, and I think they're compatible in function with an XML storage backend, if not in spirit. No, they won't map directly to nested XML tags. So what? I'd just have a self-closing tag like <sectionchange sections="section1,section2"/>. The sections any bit of topic is contained within are simply those listed in the last sectionchange tag. The sections attribute could be null or missing to indicate unsectioned blocks. This would allow you to have the sections running entirely parallel to the actual XML structure of the document, which is good when someone wants to use sections that don't translate well into well-formed nested expressions.

-- WalterMundt - 08 Feb 2004

Bumping Feature topic marked as Scheduled for CairoRelease that hasn't been modified recently.

Apparently the lines with SECTION and ENDSECTION in the code need work. They either need need having code and semantics attached or deleting.

-- SamHasler - 20 Apr 2004

Peter. As you are the Core contact for this, can you please confirm that it works (or not?) I don't seem to be able to use it on my site, and I need it now.

like the other Features that I have recently taken, and assigned to Dakar as I need them for work - I'm willing to take this work over if you like..

-- SvenDowideit - 12 May 2004

Feature bumbed to Dakaar due to lack of progress. There is also a MacrosPlugin about to be released that does parameterised includes.

-- CrawfordCurrie - 30 Jun 2004

See also MultiStartStopIncludes. This is a low-hanging fruit that could easily be done for DakarRelease.

-- CrawfordCurrie - 18 Jan 2005

Patch Portion Removed:

In order to work with the TWiki Cairo release (dated 9/2/2004), NAMEDINCLUDES.patch (dated 12/18/2003) needs to be tweaked. The following has been removed from the patch:

--- lib/TWiki.pm   2005-08-03 10:15:24.516017021 -0400
+++ lib/TWiki.pm   2005-08-03 09:26:19.237460728 -0400
@@ -1765,6 +1765,9 @@
             $value = $str;
         }
     }
+
+    $value =~ s/\\//g unless (($value =~ /\$pattern/) || ($name eq "pattern")); # Unescape any special chars in the format
+
     $value =~ s/\\$TranslationToken/\"/go;  # resolve \"
     return $value;
 }

-- JonDiekema - 03 Aug 2005

Implemented the %SECTION{"name"}% %ENDSECTION{"name"}% syntax for DakarRelease. Check r6434.

-- RafaelAlvarez - 16 Sep 2005