Bug: Formatted search doesn't show all results

FormattedSearch in the InternetExplorerSidebar has started to show very old links about half way down, e.g. pages from May or June, even though Codev has been busy recently. Also happens on other webs.

Test case

Click on Main.WebSidebarIEAll and compare to Codev's Changes link - even though minor changes don't show up in the Changes list, that doesn't explain the discrepancy. Not sure when this started happening, but I think it might be the last day or so - has anything changed in TWiki.org's code?

Actual SEARCH string is, with URLPARAM and HTML omitted for readability:

%SEARCH{".*" web="Codev" regex="on" nosearch="on" order="modified" reverse="on"
    nototal="on" limit="18" format=" .... actual HTML here links to $topic ....."}%

Actual results - curiously, topics such as NbspBreaksRssFeed are not present below at time of writing, but are listed in my sidebar, about 2/3 of the way down, refreshed a few minutes ago... Also, the topic InternetExplorerSidebar appears below (at time of writing) but not in the sidebar...

WebStatistics
TWikiOrgStatistics
UnregisteredUsersShouldBeTWikiGuests
PrinceXML
WebService
EscapeHtmlInTWikiRegistration
KnownIssuesOfTWiki06x01
KnownIssuesOfTWiki06x00
DownloadTWikiVM
TWikiConsultants
KnownIssuesOfTWiki04x00x00
KnownIssuesOfTWiki05x01
KnownIssuesOfTWiki05x00
KnownIssuesOfTWiki04x03
KnownIssuesOfTWiki04x01
KnownIssuesOfTWiki04x02
TWikiSecurityAlerts
DisplayOfHeadings

Environment

-- RichardDonkin - 20 Jul 2002

Follow up

The last 'recent' topic seems to be VerbatimFailsOnSomeTags, but the same problem happens on other topics so I don't think it's dependent on a certain topic. This is the tenth topic in the sidebar.

Changing the search to only look for topic names did not fix the problem. Side note: The various sidebar searches should all include scope="topic" in the SEARCH, to avoid actually searching topic text when all that's needed is the topic name.

WebChanges has a similar problem, but not identical - there are a lot of missing topics between 5th and 16th July, for example. However, http://twiki.org/cgi-bin/changes/Codev is OK.

-- RichardDonkin - 20 Jul 2002

This is probably an artifact of renaming the Test web to Sandbox, I did that on Friday. I created a small script to patch existing topics that point to the topics in the Test web, i.e. Test.SomeTopic, [[Test.SomeTopic]], [[Test.SomeTopic][some label]], web="Test.SomeTopic". The script changed the file time stamp of about 30 topics in the Codev web; search sorted by modfication date has an algorithm tuned for performance that fails in this unusual case: For performance not all topics are opened to find out the time stamp. Rather, file are sort by timestamp (to get a rough list); the list is shortened to the limit plus a slack of 20 topics; then the shortened list is sorted by rev date to get the acurate list.

So, my guess is: All should be fine after a few days.

For refereence, here is the lo of changed topics when renaming the Test to Sandbox:

• Checking web TWiki.Codev
  • AndDoesntWorkInTopicNameSearch: web="Test" s
  • AuthenticationBasedOnGroups: * Test.Test
  • AuthenticationBasedOnGroups: * Test.Test
  • BackFromPreviewLosesTextOld: ng Test.Sand
  • BugInPre: in Test.Sand
  • ConditionalRendering: [[Test.Cond
  • DefinitionListNotParsed: g (Test.RobC
  • EmailThisPageLink: [[Test.Rich
  • FormsArchive: at Test.Item
  • InternetExplorerSidebar: he Test.Side
  • LiterateDevelopment: [[Test.TWik
  • LiterateDevelopment: to Test.Perl
  • MozillaSidebar: to Test.Side
  • NbspBreaksRssFeed: [[Test.Test
  • PluginProblemWithIncludes: [[Test.Test
  • PluginProblemWithIncludes: [[Test.Twik
  • PluginProblemWithIncludes: [[Test.Twik
  • RefreshEditPage: in Test.Sand
  • RefreshEditPage: on Test.Sand
  • RefreshEditPage: ed Test.Sand
  • RefreshEditPage: ut Test.Sand
  • RefreshEditPage: in Test.Sand
  • RenameALLOWWEBCHANGEProblem: in Test.WebP
  • RenameALLOWWEBCHANGEProblem: ic Test.Next
  • RenameDoesNotCorrectSpacedTopics: [[Test.Test
  • RenameTopicDoesMainMainUser: [[Test.Clic
  • RenderMultiLineTextAreaFormField: [[Test.Test
  • RenderMultiLineTextAreaFormField: \n[[Test.Test
  • SavePreviewTextOnServer: on Test.MyTo
  • SherlockAndMozillaSearching: [[Test.Side
  • SherlockAndMozillaSearching: al Test.Side
  • SherlockAndMozillaSearching: [[Test.Side
  • SmiliesBreakRssFeed: [[Test.Rich
  • TWikiFormattingInsideCells: web="Test" f
  • TWikiFormattingInsideCells: web="Test" f
  • TWikiFormsDontExpandVariables: web="Test"}%
  • TWikiFormsDontExpandVariables: [[Test.Text
  • TWikiFormsDontExpandVariables: *[[Test.Test
  • TWikiFormsDontExpandVariables: | Test.Topi
  • TWikiFormsDontExpandVariables: a Test.Item
  • TWikiSyndication: | WebR
  • TWikiTerminology: de Test.Item
  • TWikiTerminology: [[Test.Item
  • TWikiTerminology: [[Test.Item
  • TWikiToolChest: he Test.Item
  • TWikiToolChest: he Test.Item
  • TWikiToolChest: s. Test.Item
  • TWikiToolChest: y, Test.Item
  • TWikiToolChest: [[Test.Item
  • TWikiToolChest: la Test.Item
  • TableEditor: [[Test.Item
  • VariablesCantBeSetToZero: in Test.WebP
  • VerbatimFailsOnSomeTags: ee Test.Verb
  • WikiLearnPagesOnTestWeb: web="Test" n
  • WikiLearnPagesOnTestWeb: [[Test.ForO
• Checking web TWiki.Main
  • AlanYang: \n\r\nTest.Alan
  • AlexeyCheremisin: * Test.Real
  • AmbjoernGoeransson: \n\r\nTest.Ambj
  • ArneBab: at Test.Free
  • BenjaminRabe: -\r\nTest.Sand
  • CeciliaFellenius: .. Test.Ceci
  • ChrisWolske: >\r\nTest.Chri
  • ClaytonHuntsman: * Test.Clay
  • ColdBrook: .[[Test.Cale
  • ColdBrook: >[[Test.Sett
  • CwOnline2001: [[Test.WebH
  • DaleReitsma: \n\r\nTest.Dale
  • DaleReitsma: DE Test.Proj
  • DamianCollins: \n\r\nTest.Test
  • DamianCollins: = Test.MyFo
  • DannyMuizebelt: to Test.Dann
  • DarioSartini: ++ Test.Dasa
  • DasarIndex: \r\n Test.Dasa
  • DasarIndex: b: Test.Dasa
  • DasarIndex: ks Test.Dasa
  • DasarIndex: ty Test.Dasa
  • DasarIndex: \r\n Test.Dasa
  • DasarIndex: s: Test.Dasa
  • DasarIndex: ws Test.Dasa
  • DasarIndex: na Test.Dasa
  • DasarIndex: io Test.Bern
  • DasarIndex: \r\n Test.Dasa
  • DasarIndex: y: Test.Ange
  • DasarIndex: eb Test.Anna
  • DasarIndex: eb Test.Barb
  • DasarIndex: eb Test.Clel
  • DasarIndex: eb Test.Gemm
  • DasarIndex: eb Test.Mira
  • DasarIndex: eb Test.Paol
  • DasarIndex: eb Test.Robe
  • DasarIndex: eb Test.Ralp
  • DasarIndex: eb Test.Sand
  • DasarIndex: eb Test.Serg
  • DasarIndex: eb Test.Toti
  • DasarIndex: eb Test.Dari
  • DasarIndex: \r\n Test.Dasa
  • DasarIndex: s: Test.Dasa
  • DasarIndex: to Test.Dasa
  • DasarIndex: te Test.Dasa
  • DasarIndex: ni Test.Dasa
  • DasarIndex: s: Test.Dasa
  • DasarIndex: e: Test.ItsM
  • DasarIndex: ++ Test.Dasa
  • DasarIndex: st Test.Dasa
  • DaveWilliamson: \r\n Test.Test
  • DirkHerreman: _\r\nTest.Clic
  • DuccioVigolo: - * Test.Ducc
  • DuccioVigolo: - * Test.Ducc
  • DuccioVigolo: - * Test.Demo
  • ECalendarGroup: -\r\nTest.ECal
  • EmmaJane: s: Test.Xtri
  • EricE: is Test.Nine
  • FsItGroup: [[Test.FsIt
  • GrahamKing: ng Test.NotF
  • HugoHansen: om Test.MitE
  • IngolfMagnus: * Test.Solu
  • IsabelleAgrer: k\r\nTest.Isab
  • JasonZheng: * Test.Jaso
  • JennyLorenzo: * Test.Sche
  • JennyLorenzo: * Test.Post
  • JimFirby: * Test.JimF
  • JoeLombardo: -> Test.JoeL
  • JoeTennis: -\r\nTest.JoeT
  • JohnRouillard: b. Test.NewT
  • JonAllen: \n\r\nTest.JonA
  • JonDonahue: \n\r\nTest.JonD
  • JosephSchlesinger: 'm Test.Just
  • KonstantinIgnatyev: [[Test.Grou
  • KonstantinIgnatyev: * Test.WhyI
  • KonstantinIgnatyev: * Test.Stra
  • LeeMarzke: ic Test.Test
  • MahendraBabu: * Test.Home
  • ManuBilli: ur Test.AvBa
  • MarcoCarnut: ent="Test" d
  • MarkRussell: * Test.Mark
  • MartinKopser: d: Test.Cool
  • MaxSalazar: he Test.TuBy
  • MaxSalazar: u, Test.TuBy
  • NticStapsGroup: et Test.AvBa
  • PaulDougan: e="Test.Sand
  • PierreNarvor: is Test.Pier
  • RameshNagarajan: nk Test.MyIm
  • RichGraves: \n\r\nTest.WebC
  • RichStuber: d, Test.NotF
  • RickLavin: \n\r\nTest.Rick
  • RobCampbell: * Test.RobC
  • RoddyMacFarquhar: \n\r\nTest.Rodd
  • RtsoftCo: or Test.Real
  • SalveJNilsen: n (Test.Sand
  • SamiMaatta: Q> Test.Some
  • SigridEk: \n\r\nTest.Sigr
  • TedRomer: \n\r\nTest.TedR
  • ThomasDietmaier: or Test.Link
  • TimDunn: \n\r\nTest.TimD
  • TindyHellman: \n\r\nTest.Mina
  • TindyHellman: \n\r\nTest.Elij
  • TindyHellman: \n\r\nTest.Sail
  • TorbenGB: e: Test.Torb
  • TorbenGB: nd Test.Torb
  • UrsRau: g: Test.UrsR
  • UweSchultes: \r\n Test.IstL
  • YaldremMajeed: s: Test.Clic
  • YoNo: -) Test.Worl
  • ZviBoshernitzan: * Test.TclC
• Checking web TWiki.Plugins
  • CommentPlugin: [[Test.WebH
  • FormPivotPlugin: [[Test.Test
  • FormPivotPlugin: t][Test.Test
  • SimplerPluginInstallation: [[Test.WebP
  • SimplerPluginInstallation: [[Test.WebP
  • SpreadSheetPlugin: [[Test.Spre
  • SpreadSheetPluginDev: he Test.Spre
  • TWikiDrawApplet: * Test.Test
  • TWikiDrawApplet: on Test.Test
  • TablePluginDev: [[Test.WebH
• Checking web TWiki.Support
  • AttachFileFailureMalformed: [[Test.WebH
  • BinaryFileProblem: le Test.Test
  • FormattedSearchProblem: \n[[Test.Test
  • FormsAndTemplates: he Test.Item
  • FormsAndTemplates: he Test.Item
  • FormsAndTemplates: he Test.Item
  • FormsIgnoredByInclude: he Test.Item
  • FormsIgnoredByInclude: ed Test.Item
  • HowToAddTodaysDateInfoToForm: in Test.Item
  • HowToAddTodaysDateInfoToForm: he Test.Item
  • HowToDoBookReviews: [[Test.Item
  • IncludeAndDrawingWontWork: [[Test.Twik
  • MaintainHistoryOfQueries: [[Test.Quer
  • OperaPreviewProblem: [[Test.WebH
  • OptionalCreateNewDirectoryNewTopic: * Test.Item
  • RefreshAsSecurityProblem: \n\r\nTest.Show
  • SearchResultOrderedByFormfield: web="Test" s
  • SearchResultOrderedByFormfield: of Test.Item
  • TableMaintenaceViaForm: [[Test.Item
  • TopicSaveErrorOnISPServer: le Test.Test
  • WebIncludes: ge Test.Late
• Checking web TWiki.TWiki
  • SpreadSheetPlugin: [[Test.Spre
  • TWikiAdminCookBook: [[Test.Item
  • TWikiGlossary: [[Test.Anot
  • TWikiOrgTopicIndex: web="Test"}%
  • TWikiTutorial: ke Test.Test
  • TWikiTutorialIllustrated: ke Test.Test
  • WelcomeGuest: [[Test.WebH
• Checking web TWiki.Tax
  • WebSearch: lue="Test">T
• Checking web TWiki.Test
  • ActKnowledge: o="Test.ActK
  • Sandbox.Alpha: o="Test.Alph
  • AmbjoernGoeranssonPictureGallery: o="Test.Ambj
  • AnnaWeb: e="Test.Dasa
  • AnnaWeb: o="Test.Anna
  • AnotherTopicLink: o="Test.Anot
  • AshLetMeChange: o="Test.AshL
  • AvBase: o="Test.AvBa
  • BilboBaggins: o="Test.Bilb
  • CICFConstructionGroup: er Test.Sand
  • CICFConstructionGroup: o="Test.CICF
  • Sandbox.Calendar: o="Test.Cale
  • CeciliaFelleniusPictureGallery: o="Test.Ceci
  • CleliaWeb: o="Test.Clel
  • ClickHere: o="Test.Clic
  • ClickHereToo: o="Test.Clic
  • CoolLink: o="Test.Cool
  • CulturalTransformations: o="Test.Cult
  • DasarArte: e="Test.Dasa
  • DasarArte: o="Test.Dasa
  • DasarCommunity: * Test.Ange
  • DasarCommunity: * Test.Anna
  • DasarCommunity: * Test.Barb
  • DasarCommunity: * Test.Clel
  • DasarCommunity: * Test.Dani
  • DasarCommunity: * Test.Fran
  • DasarCommunity: * Test.Gemm
  • DasarCommunity: * Test.LeoW
  • DasarCommunity: * Test.Mira
  • DasarCommunity: * Test.Paol
  • DasarCommunity: * Test.Robe
  • DasarCommunity: * Test.Ralp
  • DasarCommunity: * Test.RamW
  • DasarCommunity: * Test.RodW
  • DasarCommunity: * Test.Sand
  • DasarCommunity: * Test.Serg
  • DasarCommunity: * Test.Toti
  • DasarCommunity: * Test.Dari
  • DasarCommunity: o="Test.Dasa
  • DasarFiles: e="Test.Dasa
  • DasarFiles: o="Test.Dasa
  • DasarLinks: e="Test.Dasa
  • DasarLinks: * Test.Dasa
  • DasarLinks: * Test.Dasa
  • DasarLinks: * Test.Dasa
  • DasarLinks: * Test.Dasa
  • DasarLinks: * Test.Dasa
  • DasarLinks: * Test.Dasa
  • DasarLinks: * Test.Bern
  • DasarLinks: o="Test.Dasa
  • DasarMaterials: e="Test.Dasa
  • DasarMaterials: o="Test.Dasa
  • DasarSuoni: e="Test.Dasa
  • DasarSuoni: o="Test.Dasa
  • DasarWeb: o="Test.Dasa
  • DbTestPlans: \n\r\nTest.Logi
  • DbTestPlans: o="Test.DbTe
  • DuccioVigoloTest: e: Test.Pagi
  • DuccioVigoloTest: *: Test.Pagi
  • DuccioVigoloTest: vo Test.Pagi
  • ElijahWood: o="Test.Elij
  • EngineeringGroupGoals: o="Test.Engi
  • EnterTables: o="Test.Ente
  • ExcelCourse: o="Test.Exce
  • ExcelIntro: e="Test.Exce
  • ExcelIntro: o="Test.Exce
  • FcoJRoncero: = Test.FcoJ
  • FcoJRoncero: = Test.FcoJ
  • FcoJRoncero: o="Test.FcoJ
  • GoF: e="Test.Home
  • GoF: o="Test.GoF"
  • GroupHQC: te Test.Some
  • HeadingsAsAnchors: o="Test.Head
  • HelloWorld: \n\r\nTest.Brea
  • HomePage: \n\r\nTest.MyQu
  • HomePage: o="Test.Home
  • HvorGaarDenneLinken: o="Test.Hvor
  • IanGordonTestWeb: o="Test.IanG
  • InformationTechnologyVolunteering: o="Test.Info
  • InventarioCD: o="Test.Inve
  • IsabelleAgrerPictureGallery: o="Test.Isab
  • IstLink: o="Test.IstL
  • ItemDoesThisWork: lue="Test"}%
  • ItemRenameFromMainWeb: to Test.TkWa
  • ItemTestOfToDoForm: o="Test.Item
  • ItemToDoForm: m="Test.ToDo
  • ItemToDoForm: o="Test.Item
  • JackJones: o="Test.Jack
  • JasonTest: o="Test.Jaso
  • JasonZheng1: is Test.Jaso
  • JasonZheng1: o="Test.Jaso
  • JoelhassoBookmarks: o="Test.Joel
  • JoelhassoExperimental: e\r\nTest.Joel
  • JoelhassoExperimental: o="Test.Joel
  • JoelhassoExperimental2: o="Test.Joel
  • JoelhassoHome: [[Test.Joel
  • JoelhassoHome: [[Test.Joel
  • JoelhassoHome: o="Test.Joel
  • JohnSmith: e="Test.Just
  • JohnSmith: o="Test.John
  • JonDonahueTrash: o="Test.JonD
  • JuanRoncero: = Test.FcoJ
  • JuanRoncero: = Test.FcoJ
  • JuanRoncero: y="Test.FcoJ
  • JuanRoncero: o="Test.Juan
  • JuanRoncero: r="Test.FcoJ
  • JuanRoncero: o="Test.Juan
  • JustTesting: to Test.Just
  • JustTesting: o="Test.Just
  • LifeLight: o="Test.Life
  • LigaAntonioVega: lue="Test"}%
  • LigaAntonioVega: o="Test.Liga
  • LineOfText: ou Test.Ente
  • LineOfText: o="Test.Line
  • LinkCreationOnTheFly: o="Test.Link
  • LoginLogout: o="Test.Logi
  • MarkzConversationSpace: o="Test.Mark
  • MinaBilder: o="Test.Mina
  • MinaKlassKompisar: o="Test.Mina
  • MitEgetOrd: o="Test.MitE
  • MyForm: o="Test.MyFo
  • MyFriends: e="Test.Home
  • MyFriends: o="Test.MyFr
  • MyImportantInfo: o="Test.MyIm
  • MyQuestions: o="Test.MyQu
  • MySandBoxMsz1: \n[[Test.MySa
  • MySandBoxMsz1: \n[[Test.MySa
  • MySandBoxMsz1: \n[[Test.MySa
  • MySandBoxTopic: to Test.WebH
  • MySandBoxTopic: :\r\nTest.Wiki
  • NeuesTopic: lue="Test"}%
  • NinetyPercentMental: o="Test.Nine
  • Sandbox.Normal_words_like_these: o="Test.Norm
  • NotFlat: le Test.Line
  • NotFlat: o="Test.NotF
  • PaginaXYZ: a: Test.Pagi
  • PaginaY: e: Test.Ducc
  • PaginaY: \n\r\nTest.Pagi
  • PaginaZ: - * Test.Pagi
  • PaginaZ: e\r\nTest.Pagi
  • PierresOwnSandBox: o="Test.Pier
  • ProdOrder: lue="Test"}%
  • ProjectList: o="Test.Proj
  • PubReview: lue="Test">
  • Sandbox.RKEM: by Test.Torb
  • RandyNewTopic: e="Test.Rand
  • RandyNewTopic: o="Test.Rand
  • RandyStephensSandBox: o="Test.Rand
  • RealTime: o="Test.Real
  • ReportProblemsHere: o="Test.Repo
  • SailorMoon: o="Test.Sail
  • SandBox22: lue="Test"}%
  • SandBox26: lue="Test"}%
  • SandBox37: lue="Test"}%
  • SandBox60a: o="Test.Sand
  • SandBoxA: o="Test.Sand
  • Sandbox.Settings: o="Test.Sett
  • SignofKnowledge: o="Test.Sign
  • SigridEkPictureGallery: o="Test.Sigr
  • SolutioFSM: o="Test.Solu
  • SomeWords: o="Test.Some
  • SpreadSheetSandbox: m="Test.Spea
  • SpreadSheetSandbox: o="Test.Spre
  • StarTer: o="Test.Star
  • StartHere: \n\r\nTest.WebH
  • SummerIntern02: o="Test.Summ
  • TWikiRegistration: [[Test.WebH
  • TWikiWebpage: o="Test.TWik
  • TasksDue: e="Test.Home
  • TasksDue: o="Test.Task
  • TclCheatSheet: o="Test.TclC
  • TedRomerSandBox: o="Test.TedR
  • TestAllowOnlyTWikiAdminGroup: so Test.Test
  • TestCase: o="Test.Test
  • TestDenyTWikiGuest: so Test.Test
  • TestFormOut: o="Test.Test
  • TestIncludeArea: ++ Test.Test
  • TestLink1: o="Test.Test
  • TestProjectPage: o="Test.Test
  • TestTextArea: lue="Test"}%
  • TestTopic3: ent="Test" d
  • TestTopic30: o="Test.Test
  • TestTopic30: m="Test.Test
  • TestTopic32: e="Test.draw
  • TestTopic32: e="Test.gif"
  • TestTopic50: [[Test.Test
  • TestVonSven: t="Test.gif"
  • TestVonSven: e="Test.gif"
  • TextAreaFormTest: lue="Test"}%
  • ThM: lue="Test"}%
  • TorbenLinks: * Test.Torb
  • TuByTu: o="Test.TuBy
  • TuByTu2: o="Test.TuBy
  • TwikiIn: lue="Test"}%
  • TwikiWordTopicFromRandy: e="Test.Rand
  • TwikiWordTopicFromRandy: o="Test.Twik
  • VosOnVos: o="Test.VosO
  • WebCam: e="Test.Star
  • WebCam: o="Test.WebC
  • Sandbox: o="Test.WebH
  • WebPages: o="Test.WebP
  • WhyIdoNotLikeMS: o="Test.WhyI
  • WitnessList: e="Test.John
  • WitnessList: * Test.John
  • WitnessList: * Test.Jack
  • WitnessList: * Test.Bilb
  • WitnessList: o="Test.Witn
  • WorldAsPlayground: o="Test.Worl
  • WorldPeace: o="Test.Worl
• End TWiki modification, total 413 substitutions

-- PeterThoeny - 21 Jul 2002

Fix record

Seems to have fixed itself as described by Peter - however, I'm curious why the changes script didn't have this problem. Perhaps we could make the other routines use the same algorithm as this script, to make them more robust against this sort of change?

As for the script, it would be possible to make it record and re-instate the last-modified time (using the touch command or a Perl equivalent), avoiding this problem completely.

-- RichardDonkin - 22 Jul 2002

The changes script is based on the .changes file, which gets updated when saving a topic with "Minor changes" unchecked. Search must not depend on the "Minor changes" flag, only on the actual modification time of the topics. We could easily change the algorithm to look always for the RCS timestamp, but that would be a big performance loss. My quick-and-dirty hack was a one time thing. My recommendation: Keep the algorithm as it is.

-- PeterThoeny - 22 Jul 2002