SearchByDate - the easy way...

This topic started with the informal request:

Instead of searching by specific topic or string, sometimes I like to search the most recent additions or changes in a specific web. Or for that matter, search for additions or changes during a certain period.

-- CartonLao - 12 Dec 2002

after a few months discussion, which you can find by reading the previous versions of this topic, we had reached a point where we had an easy interpretation of the request, with a possible implementation, and a more involved one which was still far from complete specification. it was decided to use what we had and go on with the discussion under a different name. here we present the new 'date' parameter to a %SEARCH%, while in HistoricizedSearch we discuss a more complete possible approach to the problem.

proposal

a date parameter has been added to %SEARCH%. it is string valued and can be used to restrict the hits to the search.

definitions

if a %SEARCH% specifies a value for the 'date' parameter, a topic matching the rest of the search arguments will be returned only if its last modification date falls inside of the interval indicated by 'date'.

the date parameter will actually indicate an interval in which the date property of the topic must fall. it could be argued if the parameter should be called date or interval. all other things being equal, I like date ways better at least because it is half as short and easier to type (do you blind type?).

a string is accepted as a valid date specification if it belongs to the interval 'language':

interval ::= date
           | date '/' date
           | 'P' duration '/' date
           | date '/' 'P' duration
           ;

date ::= year ('-' month ('-' day ('T' hour (':' minute (':' second)))))
       | '$today'
       | '$now'
       ;

duration ::= NUM unit
           | duration NUM unit
           ;

unit ::= 'y' | 'm' | 'w' | 'd' | 'h' | 'M' | 's'

year, month, day, hour, minute, second ::= NUM

specification

an interval specifying only one date is to be considered as if the user had written the same string as a start and end date.

an incompletely specified date must be completed by including as much time as possible. so if a starting point is specified by the sole year, we will read 'year', January, the 1st, at zero hours zero minutes zero seconds. similarly for the ending point, December, the 31st, at 23 hours, 59 minutes and 60 seconds. I'm sorry for any lost leap seconds.

if we were using yacc, we could think of something like this...

interval ::= date                   { $$.start = fillStart($1); $$.end = fillEnd($1); }
           | date '/' date          { $$.start = fillStart($1); $$.end = fillEnd($3); }
           | 'P' duration '/' date  { $$.start = fillEnd($4)-$2; $$.end = fillEnd($4); }
           | date '/' 'P' duration  { $$.start = fillStart($1); $$.end = fillStart($1)+$4; }
           ;

if we accept that the format is to be quite rigid, the perl implementation is very easy. the rigidity lies in the fact that all fields have to be specified with a fixed width. this way it is easy completing either end with the string '0000-01-01T00:00:00' or '0000-12-31T23:59:60' respectively.

it is not too difficult to release this rigidity, but I'm not familiar enough with perl to do it... all right, I'm lazy...

examples

you write	you mean	notes
2003-12-12	on the 12th Dec 2003, from 0:00 to 23:59:59	tested ok
2003	any second in the year 2003	tested ok
2003/P20w	the first 20 weeks of the year 2003	not tested
P20w/2003	the last 20 weeks of the year 2003	not tested
2003-12-12/14	from 12th Dec 2003 0:00 to 14th Dec 2003 23:59:59	not implemented
$today	any second during the present day	tested ok
P2d/$today	today and yesterday	tested ok
P1d/$now	the last 24 hours	not tested
$now/P50y	can a topic be placed in the future?	not tested

I have the two searches in my twiki personal topic:

Topic modificati fra ieri ed oggi	%SEARCH{ "." date="P2d/$today" format="[<nop>[[$web.$topic][$topic]]] <nop>" web="Main Private Ibo Servizi Culturali" scope="text" regex="on" nosearch="on" nototal="on" }%
Topic che mi riguardano modificati recentemente	%SEARCH{ "Main\.MarioFrasca" date="P3w/$today" format="[<nop>[[$web.$topic][$topic]]] <nop>" web="Main Private Ibo Servizi Culturali" scope="text" regex="on" nosearch="on" nototal="on" }%

patches

relative to twiki 2003-02-01: refer to version 1.18 of this topic

relative to twiki 2004-09-02: refer to version 1.21 of this topic

svn 3765: SearchByDate.diff

please don't consider the attachment to this page.

---

MarioFrasca - 07 Mar 2005

Nice one, Mario! Especially the spec - solid specification is always very welcome.

DevelopBranch has introduced the new module TWiki::Time, which would seem to me to be a logical place to create a time interval object. I have an interest in you doing that, because it makes sense to use your internal spec in MailerContrib and ActionTrackerPlugin. I also don't want to see more code larded into Search.pm than absolutely necessary - that module is already overweight.

-- CrawfordCurrie - 24 Feb 2005

Mario, the change looks good but I'm not going to merge anything until documentation is available. This is quite a complex change, and the risk of releasing something without adequate documentation is too high. Documentation changes should be made in the topics in an SVN checkout and added as part of the patch.

-- CrawfordCurrie - 08 Mar 2005

Crawford, I'm not sure I understand what you mean by "documentation", I would say that this page does document the feature quite extensively. ... oops, probably I get an illumination now... you mean that what I wrote in the =pod / =cut block is not in twiki syntax. or what is the problem? can you point me to some good example of the way TWiki features are documented?

-- MarioFrasca - 08 Mar 2005

In the TWiki web of your DEVELOP checkout there is a topic called "TWikiVariables" that documents the syntax and semantics of the variables such as SEARCH (well, it tries to). That topic is checked into SVN.

While I like good pod docs, I care a lot more about good user docs, and it's much easier for the developer to update that page than have to rely on the core team to merge the documentation in this topic.

-- CrawfordCurrie - 08 Mar 2005

I'm attaching another patch (Time.pm, Search.pm, TWikiVariablesNtoZ.txt), this time including the doc. if you add a TimeInterval topic in the TWiki web, I might fill it with a more detailed description, even though I prefer following standards and including references.

I'm not so sure about the procedure, it seems a bit too complicated. all right as far as the code is concerned, but the docs, well, why not simply use twiki.org? on my system I (don't want to) have a twiki interface to the svn data dir since I share the same data dirs between different versions of twiki. This allows me to test the new engine and still safely access the same data from the last stable version. I might create a symlink in the future if the procedure stays as it is now. but then if any commit to the TWiki configuration topic on svn modifies my settings... well, I'm sure I don't want that.

-- MarioFrasca - 08 Mar 2005

It would be good to use TWiki.org, but there are a number of issues that need to be solved first:

1. DEVELOP is a branch. What is done there is not guaranteed to be in the next release of TWiki
2. There may be many branches, each with different docs
3. If you don't update docs on the branch, you can't tell without generating a release whether all the correct docs are there. And Peter is the only person who can generate a release.

I also use the same data with multiple installations, but I keep the TWiki and Main webs specific to individual installations and share all the other webs. I currnetly have athens, beijing, cairo, DEVELOP and MAIN all installed, with DEVELOP and MAIN running directly off SVN checkout areas, all sharing the same data webs.

-- CrawfordCurrie - 09 Mar 2005

all right, I see, but I still need someone to create an empty TimeInterval topic, otherwise svn diff does not take my modifications into account.

even though I may agree that the docs should be separated from version to version, I still would prefer keeping one configuration, allowing for more severe testing. I thought there was some discussion about separating config topics from the docs.

-- MarioFrasca - 09 Mar 2005

OK, it's there.

Yes, it would be good to separate docs from config. There have been some moves in that direction, but they seem to be hanging, currently.

-- CrawfordCurrie - 09 Mar 2005

Crawford, I'm looking forward to having to solve conflicts on this subject! SearchByDate3778.diff: includes TWikiVariables / TimeInterval documentation.

ciao,

-- MarioFrasca - 12 Mar 2005

finally added.

DEVELOP/data/TWiki/TimeInterval.txt
DEVELOP/lib/TWiki/Search.pm
DEVELOP/lib/TWiki/Time.pm
DEVELOP/data/TWiki/TWikiVariablesNtoZ.txt
DEVELOP/data/TWiki/TimeInterval.txt
Committed revision 3850.

-- MarioFrasca - 23 Mar 2005

(Sorry for reopening this request. Please tell meif it would be better to open a new request.)

Searching by date gives only the modified date. I'm looking for an option to search for a create date timespan. I will give one of many possible scenarios which is not possible without such a create date search feature.

The Scenario:

You want to traverse some pages by their create date, e.g. News Entries. You could place a search on this site which searches the "next" entry by searching for all entries created later ($now/P50y excluding itself), ordering them by create date (which is already possible) and reducing the result to one (limit="1"). Analogous you can search for a "previously" created entry.

What do you think?

-- TobiasRoeser - 26 Mar 2006

Ok, I opened a new Feature Request with SearchByCreateDate.

-- TobiasRoeser - 28 Mar 2006