Question

I recently installed twiki as a public website (twiki.brayden.org). It seems to be working wonderfully (thank you!!!). But today I made a hidden web - I'm trying to convince a customer that it is OK to collaborate this way, and that it's secure (enough). The authentication works great - I used the view - viewauth way with .htpasswd security. The only problem is that the search functions do not work in that web, even if I initiate the search from within that web and search for a term that I know is there - I always get 0 hits. All other features seem to work fine - including some that depend on search, such as WebChanges. I haven't spent any time at all debugging this yet - I was hoping to find an answer here.

• TWiki version: Dec 2001
• Web server: apache
• Server OS: bsd
• Web browser: all (IE5.5, Mozilla 1.0, Konqueror)
• Client OS: NT 4, Linux

-- DaleBrayden - 11 Jul 2002

Answer

That is kind of odd. At work we do have this setup and it does work: The NOSEARCHALL web preferences variable is set to on for access restricted webs. A search from an other web does not find anything in the restricted web; a search from within the same restricted web does find topics in the current web. Look at the search form, does it include the web parameter by chance?

-- PeterThoeny - 12 Jul 2002

I tried the advanced search page. The hidden web shows up in the drop-down list of webs. If I select 'all' webs for the search, the results page shows every web, including the hidden one, but all have zero results. WebChanges works, but WebIndex does not. WebTopicList works. I don't see any errors being reported - I checked the http error log and nothing unusual shows up there.

-- DaleBrayden - 12 Jul 2002

---

Finally in December my users on hidden webs began to complain about lack of search, so I dug into it a bit more. I found that in Search.pm there is a test for 'checkAccessPermission(script, ..., topic), in which script = 'view'. I changed it to be 'viewauth' and the search function now works as expected: search from within a hidden web searches that web plus all public webs. Search from outside a hidden web does not search hidden webs. Here's the revised code snippet, starting from Search.pm line 443 December 2001 version:

} else {
  # lazy query, need to do it at last
  ( $meta, $text ) = &TWiki::Store::readTopic( $thisWebName, $topic );
  # for a test - check viewauth
  $allowView = &TWiki::Access::checkAccessPermission( "viewauth", $TWiki::wikiUserName,
               $text, $topic, $thisWebName );

-- DaleBrayden - 03 Mar 2003

This is an incorrect fix, "viewauth" does actually disable the permission check. This is because TWiki will check for a Set ALLOWTOPICVIEWAUTH setting, which does not exist.

-- PeterThoeny - 04 Mar 2003

I don't know what to tell you - using the view/viewauth approach as described in TWikiAccessControl, my search did not work before I made the change, and it works now. And search from outside a hidden web does not search inside a hidden web. And a search by an unauthenticated user, who by definition is viewing a non-hidden web, works - showing results from all public non-hidden webs.

-- DaleBrayden - 04 Mar 2003

If search failes in all "selfmade" Webs: have a look at SearchDoesNotWork. This fixed my searchfailure in all selfcreated (hidden) Webs.

-- SwenSchneider - 05 Mar 2003

Thanks for that pointer. Of course, my search is working as described above. The note in SearchDoesNotWork addresses the issue where "view" requires valid-user. In my case "viewauth" requires valid-user, but "view" does not - I followed the view/viewauth way of defining some hidden/some public ways, as described in TWikiAccessControl. And I really wouldn't want "search" to require valid-user, because that would mean people would have to login to do a search even in public webs.

-- DaleBrayden - 05 Mar 2003

I've dug into the Prefs.pm module a bit - of course, the response by PeterThoeny on 04 Mar was correct - the checkAccessPermission() function knows nothing about "viewauth". The only reason my 'fix' worked was that the NOSEARCHALL setting prevents people from outside hidden webs from searching inside hidden webs. But as I traced through the code in Prefs.pm I found what I tentatively believe to be fundamental flaws in the implementation - the getPreferenceValue() and getPreferencesValue() have interesting interactions and side-effects that I believe will explain the behavior that I saw.

This general issue came up for me again on the twiki at my workplace - the twiki installation there requires authentication to get in, and has view restrictions by group on a number of the webs. This is quite different than my personal twiki (no authentication for most webs; hidden webs require authentication). I found at my workplace that search, using the vanilla Search.pm from the Dec 2001 version of twiki, was displaying search results from webs that the user did not have the right to view, based on WebPreferences ALLOWWEBVIEW settings.

As I traced the code I found that the setting for ALLOWWEBVIEW was being taken from the web in which the user initiated the search, rather than the webs being searched. I have not gotten to the exact bottom of this problem yet, but I traced through the code in Prefs.pm and I saw 1) correctable inefficiencies; 2) actual bugs where functions that can take a "web" parameter are not being passed a "web" parameter when they should be; and 3) side effects in which the FINALPREFERENCES variable is being applied prematurely and incorrectly.

I don't have my notes (or the source to which I added the tracing statements) so I can't be more specific right now. But in general I traced the points where the prefsKeys and prefsValues arrays were updated, and also the points where finalPrefsKeys was updated. The results were ... interesting and unexpected (to me).

Now - I find it very hard to believe that there are fundamental bugs in Prefs.pm - after all, there are hundreds of TWiki installations, probably. And the people who have developed and maintained TWiki use great care and have tremendous skill. But so far it looks to me that there are bugs. I will find out more next week. Has anyone else seen issues with Prefs.pm ?

-- DaleBrayden - 15 Mar 2003

Yup, I found the same problems. My guess is that read protections are not in the wiki spirit, so not much thought is done on it by the core developers. But I agree, it is very important for intranet use...

-- ColasNahaboo - 17 Mar 2003

I too am having this problem and have found a secure solution. The problem is that the CGI script "search" is usually registered with .htaccess as "allow from all" and not "require valid-user", so the user is not authenticated at time of search. If you change this to "require valid-user" then all of your public searches will break ... not a good solution. My solution is to create another search script and twiki template topic...

1)  Go to the web topic %TWIKIWEB%.WebSearch (usually TWiki.WebSearch)
2)  Edit the topic, copy the contents, then cancel the edit.
3)  Create a new topic TWiki.WebSearchAuth
4)  Paste the contents from the TWiki.WebSearch into this new topic, changing
    all references to  search%SCRIPTSUFFIX%  to  searchauth%SCRIPTSUFFIX%  .
5)  Create a new search CGI script called searchauth
   > cd twiki/bin
   > cp search searchauth
6)  Set the new script to require validation
   > vi .htaccess
# Add these lines...
<Files "searchauth">
       require valid-user
</Files>
7)  Change the search page your ALLOWWEBVIEW web to include the newly created
    topic WebSearchAuth.  eg. Edit MyWebName.WebSearch so it now contains...
%INCLUDE{"%TWIKIWEB%.WebSearchAuth"}%

This solution will allow searching of private (ALLOWWEBVIEW) web pages from within private Webs, provided the WebSearch page has been change to include TWiki.WebSearchAuth . Specifically, authenticated users will be able to search all web pages from a private Web using TWiki.WebSearchAuth, but will only be able to search all public pages from a public Web still using TWiki.WebSearch .

-- LyleWinton - 01 Jun 2004