Extend METASEARCH to support complex queries

There is no way to do even simple conjunctions in METASEARCH. If I want to search for topics that have the field "Firstname" set to "Emma" and "Surname" set to "Peel":

• %METASEARCH(name="Firstname|Surname" value="Emma|Peel"}% is pretty much useless, so I have to
• %SEARCH{"META:FIELD.name=\"Firstname\".*value=\"Emma\";META:FIELD.name=\"Surname\".*value=\"Peel\""}% which is a PITA to write, as well as wrecking Store efficiency by forcing text searching over form fields.

TWiki:Plugins.DBCacheContrib has had a syntax for doing these sorts of searches for years. It's about time TWiki supported this. DBCacheContrib supports the following operators:

The code it uses could probably be imported to core TWiki with very little effort. METASEARCH would gain the parameter query thus:

%METASEARCH{query="Firstname='Emma' AND Surname='Peel'"}%

Of course this would only support queries in a single topic, and only over the form fields, so it would be quite restricted. The DBCacheContrib / DBCachePlugin / FormQueryPlugin would still be required for full query support. But it would be a significant improvement over what's currently there.

Further, I'd like to deprecate (remember, that means "discourage" not "remove") use of %SEARCH for searching meta-data.

Tracked in Bugs:Item3934

-- Contributors: CrawfordCurrie - 23 Apr 2007

Discussion

I fully support this feature and I think it is important to implement it ASAP - i.e. 4.2 - so that we have had this out in the field for as long as possible before we later introduce a more scalable storage model.

The SEARCH including meta is not something we will get rid of for a looong time. And with a DB storage model we will probably have to provide a simulated means of maintaining compatibility with the old SEARCHes in meta. Even if such a means is slow. I have personally used SEARCHes in meta in many of my TWiki applications, but I will for sure want to convert them over a 1 year period once this feature is available. Both for clarity (the regexing in meta is at list level 8 on my Nerdometer - http://www.lavrsen.dk/twiki/bin/view/Motion/KennethsNerdoMeter) and for possible speed win.

Crawford - you did not put yourself as a CommittedDeveloper in the form below. I know the form is new (from last night). a committed developer that will drive the implementation (maybe along with others) is required for being processed under the 14-day rule or to be voted for at release meeting.

I can support the deprecation provided the wording is carefully selected in the documentation. I do not want to scare our current users so the deprecation notice in docs as well as release note must be worded so it is a message to application developers to change to the new way of searching for speed and commitment to support the current searches with a compatibility feature. Maybe a configurable one. The regulars here on Codev now have calibrated each other on the meaning of deprecated, but we will for sure shock users if we are not careful in how we give the message. But for sure the meta search must be replaced by a DB type search for form data (and other meta data as well) in order to get a scalable TWiki in future. It is a necessity.

-- KennethLavrsen - 23 Apr 2007

Crawford. At FreetownReleaseMeeting2007x04x23 there was a string support of this feature. In fact we were several that claimed that this feature may be what it takes to justify the release of a 4.2.

Will you implement it? And can it get into 4.2? We are willing to wait a few extra weeks if needed.

Now that I read the proposal again I do not understand the statement "Of course this would only support queries in a single topic"

I must assume that a search for values of specific form fields will search in and return results from multiple topics that matches the search.

-- KennethLavrsen - 23 Apr 2007

Agreed. Any feature that makes queries easier to use is a good thing. Not many people are familiar with regular expressions, they are an impediment to creating TWiki apps. I would be very careful with deprecating search, there are millions of existing users and tens of millions of existing pages out there.

I suggest to switch the priority and create first a meta data query feature for SEARCH, later for METASEARCH. This is because SEARCH is typically used by wiki apps; METASEARCH is not used that often, and it searches meta data only in one topic at a time.

For SEARCH, I suggest to create a query-meta-data-syntax that is easy to remember. A typical use case is ANDing field values together, sometimes with AND NOT. Field values may have wildcards (not regexes.) One obvious syntax is to use the existing one of FormattedSearch, e.g. $formfield(name).

<brainstorming>

%SEARCH{ "$formfield(Firstname, Emma) +$formfield(Lastname, Peel) +$formfield(City, San Jose) -$formfield(Country, Costa Rica)" type="keyword" }%

Other meta data can be supported, such as $formname(AccountForm), $username(jsmith), $parent(AccountDB), $attachments(comment, foo bar), etc.

</brainstorming>

This goes a long way in simplifying SEARCH. It also makes queries less dependent on the storage backend used.

-- PeterThoeny - 24 Apr 2007

I am a bit worried that we are overloading SEARCH both from user and coding perspective. I sort of agree with CDot that it is better to have a METASEARCH that focus on being very effective in searching in form fields and later other meta data as well. I see SEARCH as a free text search and METASEARCH as more of a database type search or SELECT WHERE .... in SQL terms

-- KennethLavrsen - 24 Apr 2007

I need a clean overview. I have decided to put Crawford in the CommittedDeveloper field since he did himself add a link to this proposal in TWikiFeature04x02 so I am assuming he will drive the implementation himself. Note that CommittedDeveloper is not a binding contract with a committed schedule. It just means - "I will try and implement it some day - if it is OK with the rest of you".

-- KennethLavrsen - 24 Apr 2007

In regards of SEARCH vs METASEARCH: SEARCH is the search feature of TWiki, hence the desire to make it consistent to search meta data as well. METASEARCH only "searches" in one topic, e.g. it is a topic query feature. E.g. you can't do something like "show me all topics who's status is "Pipeline".

Therefore I believe it would be more beneficial for creating a better TWiki application platform if we first enhance SEARCH to query meta data, then METASEARCH.

-- PeterThoeny - 08 May 2007

only "searches" in one topic?

%METASEARCH{type="field" name="CurrentState" value="[A]cceptedProposal"}%

gives

AddFinishHandler AfterCreateWebHander AllowWebCreateByUserMappingManager AnchorNamesInNonISO8859Charsets AnchorRegexnotdefinedcorrectly AutocompleteUserReports BringViewPrintCloserToView CloneTopicLinkUnderMore DefaultsForAttachmentProperties DeleteToTrashSubweb DetectObviousRecursionInPref DisableWikiWordsWithNumbers DisplayOfHeadings DontShipDotPmTopics EditorWithTWikiVariablesWizard EngineCLIRequestMethod FuncIsInList FuncRenameFile INCLUDEPreflightParameter IncludeFromDifferentCharsetSite LoggingInGMT MakeUserModifiedTemplatesMoreUpgradeResilient MakingVarVARTopicCapable MdrepoBerkeleyDB2Sereal MdrepoToUseSereal MoreAttractiveForm MoreIntuitiveMoveAndDeletionOfWeb MoveToJQuery NewAutoModeForTwisty NewTopicLinkLeadingToTemplateOptions NewWebHomeForSupport NoFollowInDependentPages NoInAllPublicWebsOption PointAndClickAccessControl RedesignedDownloadPage2009 RedirecttoOnAttach RedirecttoVariables RegisterExternalHTTPHandler RemoveInconsistencyRegardingCopyright RenderParentHandler RetrofittingWebNameInViewfile SCRIPTURLdefaultUrlHostParam SMTPUseTLS SaveFileToBeAtomic SearchResultsPagination SelfServiceMdrepoManagement SetRedirecttoByPrefVar SingleSignOnForINCLUDE SpaceOnHomepageForTwikiInc SummaryBasedOnSearchTerms SupportUpdatefromtemplateInEditSaveScripts TWikiFuncGetRevisionInfoTakingMeta TWikiStoreSaveTopicWithIgnorePermissions TWikiTimeDebug TildeForULwithoutBullets TopicCaseSensitivity TopicCreationShouldHonorTemplatePreferences UserSpecifiedFilenamesForAttachments VarVARbooleanVarISTRUE ViewTemplateBreaksLeadingMarkup ViewTildeJsmith WarnWhenClosingEditPage WebStatisticsToHaveNTopicsNAttachments

METASEARCH is TODAY a SEARCH that searches META only. Problem is that you can only provide ONE parameter so I cannot search for all topics where CurrentState is UnderInvestigation and CommittedDeveloper is CrawfordCurrie. Crawfords proposal is a very natural extension to the already existing and heavily used METASEARCH. I do however miss the formatted output capability of SEARCH. I may want to search for specific fields and output something completely different from the topic found and then I am back to geeky SEARCH in meta data.

• You are right. I am learning every day. Reading the doc it looked like it would search just one topic. -- PeterThoeny - 08 May 2007

-- KennethLavrsen - 08 May 2007

Hmmm. Seems it may be there.

See AddFormatToMeta. If it is there then it is not documented and still a deep secret.

Let us try

• AddFinishHandler
• AfterCreateWebHander
• AllowWebCreateByUserMappingManager
• AnchorNamesInNonISO8859Charsets
• AnchorRegexnotdefinedcorrectly
• AutocompleteUserReports
• BringViewPrintCloserToView
• CloneTopicLinkUnderMore
• DefaultsForAttachmentProperties
• DeleteToTrashSubweb
• DetectObviousRecursionInPref
• DisableWikiWordsWithNumbers
• DisplayOfHeadings
• DontShipDotPmTopics
• EditorWithTWikiVariablesWizard
• EngineCLIRequestMethod
• FuncIsInList
• FuncRenameFile
• INCLUDEPreflightParameter
• IncludeFromDifferentCharsetSite
• LoggingInGMT
• MakeUserModifiedTemplatesMoreUpgradeResilient
• MakingVarVARTopicCapable
• MdrepoBerkeleyDB2Sereal
• MdrepoToUseSereal
• MoreAttractiveForm
• MoreIntuitiveMoveAndDeletionOfWeb
• MoveToJQuery
• NewAutoModeForTwisty
• NewTopicLinkLeadingToTemplateOptions
• NewWebHomeForSupport
• NoFollowInDependentPages
• NoInAllPublicWebsOption
• PointAndClickAccessControl
• RedesignedDownloadPage2009
• RedirecttoOnAttach
• RedirecttoVariables
• RegisterExternalHTTPHandler
• RemoveInconsistencyRegardingCopyright
• RenderParentHandler
• RetrofittingWebNameInViewfile
• SCRIPTURLdefaultUrlHostParam
• SMTPUseTLS
• SaveFileToBeAtomic
• SearchResultsPagination
• SelfServiceMdrepoManagement
• SetRedirecttoByPrefVar
• SingleSignOnForINCLUDE
• SpaceOnHomepageForTwikiInc
• SummaryBasedOnSearchTerms
• SupportUpdatefromtemplateInEditSaveScripts
• TWikiFuncGetRevisionInfoTakingMeta
• TWikiStoreSaveTopicWithIgnorePermissions
• TWikiTimeDebug
• TildeForULwithoutBullets
• TopicCaseSensitivity
• TopicCreationShouldHonorTemplatePreferences
• UserSpecifiedFilenamesForAttachments
• VarVARbooleanVarISTRUE
• ViewTemplateBreaksLeadingMarkup
• ViewTildeJsmith
• WarnWhenClosingEditPage
• WebStatisticsToHaveNTopicsNAttachments

YES. We have formatted METASEARCH. Time to raise a bug item for yet another missing doc. A bug item I may address myself.

-- KennethLavrsen - 08 May 2007

I agree with Peter, in principle, that a single interface to search is cleaner than two, and it would be nice to be able to re-use all the results formatting mechanisms. I'm not sure why METASEARCH was invented; perhaps because the current SEARCH code is just too daunting to consider extending it. There are big chunks I still don't really understand the reason for. Extending METASEARCH is easier, and consistent with the stated purpose of METASEARCH.

-- CrawfordCurrie - 08 May 2007

We seem to have consensus to extend SEARCH, not METASEARCH. Let's set the METASEARCH enhancements to not suitable, and let's follow up in SearchWithTWikiQueryLanguage.

-- PeterThoeny - 09 May 2007

See newer proposal to add $attachment() for format parameter of SEARCH: AddAttachmentsParamToFormat.

-- PeterThoeny - 16 Mar 2008

proposal moved to AddAttachmentsVarToFormattedSearch

-- SvenDowideit - 18 Mar 2008