Feature Proposal: Regex Search with Keyword Search

Motivation

When users see a search or query form they expect keyword search because Google is the standard.

TWikiApplications frequently have a query-by-example form or a filter. Behind the scene is a SEARCH with a regular expression search. That is, the application expects that the user enters a regular expression search string.

Users should be able to enter keyword search strings in a query-by-example form or a filter.

Description

The above mentioned usability issue can be solved if the regular expression search string may contain an embedded keyword search string.

An escape sequence turns a literal search on and off.

• Turn on: __STARTKEYWORDSEARCH__
• Turn off: __ENDKEYWORDSEARCH__

Impact and Available Solutions

Documentation

TBD

Examples

(This example needs more details)

Example regex search in a SEARCH:

%SEARCH{ "aaa;bbb;ccc" ... }%

Assuming bbb is this:

META:FIELD.*?name=\"Title\".*?%URLPARAM{"Title"}%

The user is expected to enter a regex for the Title field.

Now we add the escape sequence to turn on literal search just for the Title parameter:

META:FIELD.*?name=\"Title\".*?__STARTKEYWORDSEARCH__%URLPARAM{"Title"}%__ENDKEYWORDSEARCH__

Now the user can enter a literal search string for the Title field.

Here is a concrete example: The TWikiInstallation directory has a filter form to query by Installed at organization, Type of organization, TWiki customization, Type of use and Used by/for. This is done by anding the values of the form received by URL parameters. Actual SEARCH (newlines added for clarity):

%SEARCH{
  search="META:FIELD.*?TopicClassification.*?%TOPIC%\";
META:FIELD.*?TWikiInstalledAt.*?%URLPARAM{"qAt"}%;
META:FIELD.*?TypeOfOrganization.*?%URLPARAM{"qOrg"}%;
META:FIELD.*?TWikiCustomization.*?%URLPARAM{"qCus"}%;
META:FIELD.*?TypeOfUse.*?%URLPARAM{"qUse"}%;
META:FIELD.*?UsedByUsedFor.*?%URLPARAM{"qFor"}%"
  topic="TWikiInstallation*"
  nosearch="on"
  ...
}%

Currently, the user is expected to type a regular expression into the "Installed at" and "Used by/for" query fields. A user friendly search that combines regex search with keyword search allows users to enter plain text into the filters. New SEARCH:

%SEARCH{
  search="META:FIELD.*?TopicClassification.*?%TOPIC%\";
META:FIELD.*?TWikiInstalledAt.*?__STARTKEYWORDSEARCH__%URLPARAM{"qAt"}%__ENDKEYWORDSEARCH__;
META:FIELD.*?TypeOfOrganization.*?__STARTKEYWORDSEARCH__%URLPARAM{"qOrg"}%__ENDKEYWORDSEARCH__;
META:FIELD.*?TWikiCustomization.*?__STARTKEYWORDSEARCH__%URLPARAM{"qCus"}%__ENDKEYWORDSEARCH__;
META:FIELD.*?TypeOfUse.*?__STARTKEYWORDSEARCH__%URLPARAM{"qUse"}%__ENDKEYWORDSEARCH__;
META:FIELD.*?UsedByUsedFor.*?__STARTKEYWORDSEARCH__%URLPARAM{"qFor"}%__ENDKEYWORDSEARCH__"
  topic="TWikiInstallation*"
  nosearch="on"
  ...
}%

Implementation

TBD

-- PeterThoeny - 18 Jan 2005

Discussion:

What are good and intuitive names for the escape sequences?

-- PeterThoeny - 18 Jan 2005

That's a pretty horrible solution, isn't it? Why can't you set the type option in the %SEARCH from a select ?

• Horrible or not, the issue at hand cannot be solved with a type option because you need to have control over which part of the search string is a regex search, and which part is keyword search. -- PeterThoeny - 02 Sep 2006

-- CrawfordCurrie - 14 Apr 2005

I would like to bump this feature request back up. In creating a TWikiApplication, I would like to provide users with a keyword search of a subset of topics I can currently only define by a regex search. Regarding the aesthetics of Peter's solution, I'm not qualified to judge, but it would provide exactly the functionality I need.

I'm not sure what you're suggesting, Crawford. Currently, the type option requires one to choose either/or, and we're saying there's times we need both.

-- LynnwoodBrown - 10 Aug 2006

Agreed, both is needed to build customer focused applications. Several times I could have made my apps more user friendly if this is available: I need a regex search to make my app work, but who wants to burden the users with regular expressions that are part of the composed search! May be now is the time to get some constructive feedback to my proposal?

-- PeterThoeny - 10 Aug 2006

Just bumping this feature request back up to see if we can get some more discussion.

-- LynnwoodBrown - 15 Aug 2006

I am looking forward to see some creative suggestions for the escape sequence to switch between the modes.

-- PeterThoeny - 16 Aug 2006

Ah, I misunderstood what you meant. I'm still not totally sure from the examples above.

You want to be able to get from a keyword search string entered by the user, say eggs bacon bread to a regular expression, say (eggs|bacon|bread), yes?

Why can't you use SpreadSheetPlugin magic to do that?

-- CrawfordCurrie - 22 Aug 2006

I was taking with Crawford some more about this on irc and he pointed out that it is possible to do essentially what I need by a search within a search to define the topic attribute, like this:

%SEARCH{"keywords" type="keyword" topic="%SEARCH{"[T]opicClassification.*\=.*[F]eatureRequest" type="regex" nonoise="on" format="$topic" separator=", "}%"}%

Rather to my surprise, this works. Crawford created a proof of concept in NestedKeywordSearch. I'm not sure about whether it addresses all use cases you see, Peter, nor am I sure about the performance considerations. But at least it gives me something to work with.

-- LynnwoodBrown - 22 Aug 2006

The nested search can be used in some cases, keep in mind though that this is slow. A nested SEARCH cannot be used for a typical query by example form, such as the filter in the TWikiInstallation directory. I updated above example to be more clear.

Any ideas for an intuitive syntax? If we do not find one I suggest we go ahead with the implementation, using the __STARTKEYWORDSEARCH__ and __ENDKEYWORDSEARCH__ tokens to switch between the modes.

-- PeterThoeny - 02 Sep 2006

This is a great feature. My users had to search with regex for awhile. But most of them just don't get it and want the search syntax more simplified. This is the solution and I'm very excited to implement it. Thanks Peter.

-- ThomasEsau - 26 Sep 2008

I think its quite likely that this idea is covered by the 4.2 released feature - QuerySearch.

-- SvenDowideit - 29 Sep 2008

I agree with Sven. Looking at the original intention of this proposal, the Query search meets those needs and does it even nicer.

Also the query search allows a much more optimal way to later implement searches in indexed storage which is where we are moving for 5.0.

-- KennethLavrsen - 02 Oct 2008

Actually, the query search does not solve the problem, it just shifts it. I can use a query search instead of a regex search, but I still cannot AND together keyword search terms received via %URLPARAM{"..."}% within a query search. That is, I still cannot construct a query by example form with keyword search strings, e.g. we have to tell the layman to "please watch out not to use TWiki's query search syntax or you get unexpected results". So, this is still a feature request to make TWiki more user friendly. ThomasEsau understands the issue, may be he can explain what I fail to get across.

-- PeterThoeny - 03 Oct 2008

In that case, it sounds like this proposal needs more specification love, so that it manages to catch the attentions of someone that will do the implementation.

Peter? Thomas?

-- SvenDowideit - 04 Oct 2008

Pinging this feature. Needed for DontMakeMeThink user-friendly applications.

-- PeterThoeny - 2010-07-29