Feature: Create Auto-incrementing Topic Names with AUTOINC<n>

Motivation

Flexible way to generate auto-incremented topic names when creating new topics. This is used for trackers of all sorts.

Description and Documentation

When you add AUTOINC<n> in a topic name it gets replaced with an auto-incremented number on topic save. <n> is a number starting from 0, and may include leading zeros. Leading zeros are used to zero-pad numbers so that auto-incremented topic names sort properly. Deleted topics are not re-used to ensure uniqueness of topic names. That is, the auto-incremented number is always higher than the existing ones, even if there are gaps in the number sequence.

Examples:

• BugAUTOINC0 - creates topic names Bug0, Bug1, Bug2, ... (does not sort properly; same as BugXXXXXXXXXX)
• ItemAUTOINC0000 - creates topic names Item0000, Item0001, Item0002, ... (sorts properly up to 9999)
• DocTXAUTOINC1001 - start with DocTX1001, DocTX1002, ... (sorts properly up to 9999; auto-links)

Note: This is similar to TWiki's existing ItemXXXXXXXXXX keyword feature, where the save script replaces 10 or more Xes with an auto-incremented number starting from 0. This is documented in AllowDynamicTopicNameCreation and TWiki04.TWikiTemplates#Automatically_Generated_Topicnam. This feature has several limitations:

• It is not possible to create topics with an X before the number, such as DocTX1, DocTX2, etc.
• Sort of topic names does not work properly, e.g. they sort as Item0, Item1, Item11, Item12 ... Item1999, Item2, Item21, ....
• It is not possible to create unique names, e.g. names of removed topics are re-used
• Performance issue if there are 5000 or more topics
  • see below - SD

The improved AUTOINC<n> keyword feature replaced the XXXXXXXXXX feature; the latter feature is retained as an undocumented feature for backward compatibility.

Impact and Available Solutions

Implementation

This is implemented and is ready for merge.

• AUTOINC0.patch: AUTOINC0 patch for lib/TWiki/UI/Save.pm

-- Contributors: PeterThoeny

Discussion

I implemented this feature because I needed it for a TWiki application at a client's site. Topic names had to sort properly, and the application had several types of IDs identified by the topic name (DocQA0001, DocSA0001, DocTX0001, etc.)

-- PeterThoeny - 19 Jul 2006

Hm, if we are about to see an enhancement in this area I'd like to add my 2c: how about appending an autogenerated id only if there is a name clash. This would help TopicDisplayNames too.

-- MichaelDaum - 19 Jul 2006

dreamon Why not incorporate some of the features from GuidPlugin (formerly UniqueIdPlugin) and MrdIdPlugin and integrate it to the Core so that TopicDisplayNames and PermaLinks become possible? dreamoff

-- FranzJosefSilli - 19 Jul 2006

Two enhancements would come in handy:

1. Avoid race conditions (two processes get the same topic name) by locking
   • see below - SD
2. Allow autoinc topic name creation outside the save routine. In the BenchmarkFramework routines I'd like to create an autoinc topic based on the topic which is measured, and then add the raw measurement data to it. But save never returns, it simply redirects to view.

-- HaraldJoerg - 19 Jul 2006

An extra enhacement related to the TopicXXXXXXXXXX feature is to abstract the topic name generation from the TWiki::UI::Save::save() method. This way, a plugin that wants to save topics with autoincremented names can just use TWiki::Func::saveTopic() passing a $topic like 'MyPluginAUTOINC000000'. This currently is not supported by the TopicXXXXXXXXXX feature.

-- AntonioTerceiro - 25 Jul 2006

This topic never reached a concensus on the spec but all agree to extend the feature.

To push this into 4.1 we need.

• An agreed spec.
• A committed owner.

Otherwise this gets deferred to 4.2. But I do not see any protests against the feature as such. It is only the spec that is discussed.

-- KennethLavrsen - 27 Sep 2006

yep, we could do with this being extended, though I would like to point out that "Performance issue if there are 5000 or more topics" is untrue. The 10X feature stores the last highest used value, and increments from there to find the next unused number, and it uses twiki topic locking to avoid the race condition of more than one session saving at the same time.

-- SvenDowideit - 27 Sep 2006

Hmmm... What version are you referring to? Is there something in DEVELOP or develop.twiki.org that ought to be merged? In TWikiRelease04x00 SVN r11611 I could neither find a store for the highest used value nor topic locking outside of the "naked" TWiki::Store::sub _noHandlersSave(), whereas the 10X to number calculation is made a long time before that, in TWiki::UI::Save::save().

• I just double-checked: The current XXXXX related code does a one by one $n++ iteration with a $store->topicExists(), starting at 0. -- PeterThoeny - 01 Oct 2006
  • Thats incredibly bad news because I know I wrote the code to store the last used $n in a private working area. -- SD

-- HaraldJoerg - 27 Sep 2006

Pardon me, but I do not see that we did not reach consensus:

• There is a working solution (see patch attached below), ready to put into SVN. (I did not put it into the TWiki 4 branch because I wanted to wait until the TWikiRelease04x01Process was defined)
• There have not been any objections to this feature.
• There have been a number of suggestions for further enhancements.

I prefer to do one step at a time. I do not like to see waisted effort of a feature already in use by at least two production TWiki sites; if the new spec is acceptable I will check it in in the next few days.

-- PeterThoeny - 30 Sep 2006

OK so we have a committed owner I see. That was not clear because the proposals had not been addressed. If people propose enhancements or changes and they remain undisputed then it is hard to see the consensus.

The questions that I evaluated relate to the spec - and that needs to be addressed - are:

• Michael: How about appending an autogenerated id only if there is a name clash?
  • This is a further enhancement. Idea for syntax? -- PTh
• Harald: Avoid race conditions (two processes get the same topic name) by locking. Sven has later assured us that the XXXXXXXXXX feature had a locking. We need confirmation from Peter that the new feature has no race condition problem either.
  • See below. -- PTh
• Harald: Allow autoinc topic name creation outside the save routine.
  • Sounds like this could be added some later time and that the current proposal does not prevent this.
  • Agreed, this is a further enhancement and should be treated as such. -- PTh

Antonio and Franz Josef had comments that do not relate directly to the proposed feature.

-- KennethLavrsen - 30 Sep 2006

I would like to comment Michaels question: "How about appending an autogenerated id only if there is a name clash?"

The feature as proposed by Peter addresses the need to create unique topic names suffixed number with a predictable format. I think Michael's question tries to solve a quite different problem.

And my comment to Peter's proposal. The feature as such is great and I will personally use it in TWiki Applications where exactly the sorting issue of the current XXXX feature has prevented me from using it. Instead I have used a ugly date based scheme with YYYYMMDDMMSS.

It is important that the XXXXX feature remains for backwards compatibility - and that seems to be addressed also.

The only concern I have is the race condition issue which Peter may assure us is covered.

So I welcome this enhancement very warmly.

-- KennethLavrsen - 30 Sep 2006

On race condition: Both implementations have a race condition if two processes search for the next free number at exactly the same time. Although not measured, I believe the AUTOINC<n> is a bit less likely to get into a race condition for a large topic set because of shorter execution time.

BTW, other places in TWiki are not immune to race conditions either, although it is not very likely to occure. For example, a file based semaphore is not immune to race conditions, whereas a "create directory and check the result", or use Fcntl is (assuming filesystem is not NFS).

So I believe the current AUTOINC<n> implementation is "good enough" for TWiki. I modified WhatIsIn04x01 to reflect this.

-- PeterThoeny - 01 Oct 2006

This is now implemented, SVN 11628.

-- PeterThoeny - 01 Oct 2006

Thanks for making this feature Peter. I know we will start using this feature the minute 4.1 is released.

-- KennethLavrsen - 01 Oct 2006

How about allowing multiple topic pages to share the same AUTOINC counter?

E.g. I want this feature for an issue tracking system (appears common). I would LIKE to have names of the form Issue7676-SomeSimpleSubject. I.e. I want to have text in addition to the prefix that indicates the sequence number.

• hmm. I am assuming that FooAUTOINC0 and BarAUTOINC0 will have difference sequence number spaces. Do they?
  • Yes, you can have as many number spaces as you want, defined by different prefixes. -- PTh

Personal reason: a few years ago Intel IT went to a bug tracking system that just has Item numbers. No subjects. Ordinary users cannot even generate a view that looks at Item number + first line of text, although sysadmins can. I have spent far too much time clicking on every open issue, one by one, until I find the one I want.

Yes, a good view produced by search helps. Nevertheless, I would like to see [[Item54343-The Title Bug]] rather than Item54343. And both are better than Item 54343 - The Title Bug, since the latter will not have the descriptive text updated if a rename happens.

yes, I hate having [[Item76767-More stuff]] rather than a true wikiword. But Item76767MoreStuff is just too ugly.

By the way, for reasons such as these I am going towards [[IssueYYYYMMDDSS-XXXXXXXXXX-...]] - i.e. using a timestamp as the main number, with autoincrement just to encourage uniqueness.

• Matter of personal taste? For bug trackers I prefer short unique identifiers and reports that show meaningful summaries -- PTh

-- AndyGlew - 02 Oct 2006

I refactored out the SearchWithRegexSort discussion.

-- PeterThoeny - 05 Oct 2006