Feature Proposal: Support access to arbitrary meta-data (using the existing addressing schemes in IF and QuerySearch)

Motivation

In a IRC conversation with EricSorton and SvenDowideit yesterday, we discussed how SEARCH results could display meta-data from the hit topic. Obviously $formfield() exists to support access to form fields, but there is no generic mechanism.

It would be really neat if there was a $meta() tag available in SEARCH format="", that used the content access syntax of a SEARCH query to access meta-data.

Description and Documentation

For example,

| *Firstname* | *Surname* | *Age* | *Stage* |
%SEARCH{"Firstname='Emma' OR Firstname='John') AND Lastname='Peel'" type="query" format="| $meta(Firstname) | $meta(Lastname) | $meta(HistoryForm.Age) | $meta(preferences[name='WORKFLOW'].value) |"}%=

Obviously $meta(Firstname) is synonymous with $formfield(Firstname), but because $meta is so much more flexible (it has access to all of the meta-data, and can even indirect via references to other topics) it makes sense to deprecate $formfield in favour of $meta. A number of other operators that exist in search formats could be deprecated at the same time.

Another consideration that format modifiers, such as $topic(20), should be done functionally, so it's a no-brainer to support format="$trunc(20, $meta(Firstname))"

Impact

Implementation

-- Contributors: CrawfordCurrie - 20 May 2008

Discussion

Why deprecate a simple syntax in favor of a more complex?

meta is OK. But I would like to keep formfield and not deprecate it. It is a good feature with a proper name. We do not always have to deprecate everything just because we make something new. We never remove it anyway because removing means pissing off a lot of people. So why mark so many things deprecated? It is just negative biased verbal noise.

-- KennethLavrsen - 20 May 2008

No need for such strong language; your message is clear without it. My main rationale for deprecation is to avoid having to carry the baggage of duplicated code forward.

How about thinking about the positive side e.g. can the proposed syntax do everything you would want it to do? For example, can you extract a date from a TOPICINFO in a meaningful way? How can the other heuristic presentation operators (e.g. $summary ) be coded functionally?

-- CrawfordCurrie - 21 May 2008

Using IF more and more makes it painfully clear that we need a way to display the same values as we test in IF. The longhand version could be to write

%IF{"1=1" then="fields[name='Testing'].value"}%

%SEARCH{"name = Web.Topic" type="query" format="$topic - $meta(fields[name='Testing'].title) == $meta(fields[name='Testing'].value)"}%

%META{"$meta(fields[name='Testing'].value)"}%

$meta("fields[name='Testing'].value)

-- SvenDowideit - 21 May 2008

Overall I like this proposal. Also useful for plugin authors who create custom meta data.

But I am with Kenneth, we cannot deprecate $formfield(), the name is much more descriptive than the nerdy $meta() (could be my cat's name), and we cannot neglect the millions of users out there.

So, I am supportive of this proposal without deprecating anything of the existing syntax.

(As a sidenote I find the current QuerySearch syntax somewhat unfortunate because it makes it impossible to come up with a nicely designed syntax that makes it easy to mentally switch between the "search" parameter and the format="" parameter. Imagine a "$parent() = 'SupportAccessToArbitraryMetaDataFromSEARCH'" search and a format="$parent()" formatting, or a "$formfield(Food) = 'Sushi'" search and a format="$formfield(Food)" formatting. Anyway, we can't deprecate the query search syntax.)

-- PeterThoeny - 21 May 2008

Peter, it sounds like you are trying to pervert the existing $format syntax to serve in the role of a query syntax? Yes, that's probably not impossible; IIRC this, and other syntax options, were discussed at painful length during the design of the query syntax. Note that the existing format tags only support recovery of a small subset of the meta-data, and you would still need to invent something as powerful as the generalised query syntax to approach the solution we already have.

Fortunately a clean syntax extension that uses the generalised query syntax as a content access syntax in format parameters is perfectly possible, so don't get discouraged. I have to agree about the $meta name - it stinks - but I was wary of overloading $formfield, because of the pre-existing semantics. I am also wary of overlaps with SpreadSheetPlugin, which already implements many of the functions that are required for formatting search results.

Sven, thanks for the note about ExtractAndCentralizeFormattingRefactor. This feature should not be undertaken without careful consideration of that refactoring.

-- CrawfordCurrie - 21 May 2008

On syntax, I suggest to make it totally transparent so that any meta data can be queried, also those introduced by a plugin. Examples:

With this syntax, it is possible to implement the formatter generically, e.g. simply by upper-casing the parameter to construct the name of the meta data.

And for simplicity, please no shortcuts for $meta(). We already have distinct functions such as $formfield().

-- PeterThoeny - 21 May 2008

Supporting format="$trunc(20, $meta(...))" is a good idea. I suggest to open a new feature proposal for $trunc(). (already done: AddTruncateVAR - SD)

-- PeterThoeny - 21 May 2008

Thanks, Peter, for the excellent examples; it's important to test out the feasibility of an idea before putting too much effort in. After all the work done to put the shortcuts into the parser, I suspect it would be more work to take them out again than just to leave them, though I agree that there is an uncomfortable overlap with $formfield.

One question that troubles my mind is what to do about "array" and "hash" type results. $meta(field[name='Category'].value) is single-valued, but $meta(field[value='Open'].name) is potentially multi-valued, as several fields may have a value of 'Open'. In the query context this is dealt with by operating on the result sets, but in the case of a SEARCH format, how should such an array result be represented? One way would be to stringify it to a JSON list. Hashes could be represented the same way - for example, $meta(topicinfo) as { author: "VickiBrown", date: "1211294162", format: 1.1 version: 1.1 }. The use of JSON syntax simplifies REST applications.

-- CrawfordCurrie - 22 May 2008

That means that $meta(form) should also return a JSON expression holding all of the form. In general this should be the same for any $meta() that does not point to a single string.

-- MichaelDaum - 22 May 2008

How would the JSON syntax work if I simply need one parameter value, such as the equivalent of $meta(preference[name='VIEW_TEMPLATE'].value ?

-- PeterThoeny - 23 May 2008

Border case to consider with multiple="on" search. For example, what if I'd like to search for all .jpg attachments and show a neatly formatted table with image in first column and names, comments and authors in second column?

-- PeterThoeny - 23 May 2008

JSON would only be used for the case where there are multiple values to be returned.

In don't think the power is present to support presentation of array values in a TWiki table. It would require something like a foreach operator, wouldn't it? Then we are really starting to encroach on SpreadSheetPlugin.

-- CrawfordCurrie - 23 May 2008

No, with multiple="on" you already can search, say, a table in a TWiki page, apply a regex on each table row and show a re-formatted table row. I have done that for a specific TWiki application. The same could be done for meta data. probably out of scope of this proposal, but we should plan the syntax for this.

-- PeterThoeny - 23 May 2008

I understood what you meant, but something like $meta(attachments[name='*.jpg']) goes further than multiple="on" because it returns an array of results (all .jpg attached to the topic) even when multiple is not used. For a fully general solution, Sven's (offline) suggestion of embedding Javascript makes more and more sense when you think about this (indeed, it's the solution that Jot Spot used). As you say, such a fully general solution is beyond the scope of this proposal.

In terms of the constrained solution, things start to get interesting. Let's say we do have a foreach operator. Then for your example of presenting .jpgs attached to a matched topic, you might use something like this: $foreach(attachments[name='*.jpg'],'|[[$percntATTACHURL$percnt/$name]]|$name,$comment,$author|$n'). In this case $foreach is applied to an array, and I have used $name, $comment and $author to refer to the keys of the hash at each position in this array. This same $name technique could be used to refer to the fields of any hash within the scope of a $meta statement, e.g. $meta(info,'$author changed me on $n2d($date)').

Possibly a step too far for this proposal, but an indicator of a possible way ahead.

-- CrawfordCurrie - 24 May 2008