Feature Proposal: Statistics enhancements for ReadOnlyAndMirrorWebs

Motivation

If you have ReadOnlyAndMirrorWebs, you face several issues otherwise you don't face.

Employing ReadOnlyAndMirrorWebs implies that the TWiki site might have quite a few webs and a large number of page views. If there are a lot of page views (multi-million), invoking the statistics script from browser is not practical.

So here, I'm focusing on invoking the statistics script from command line or a cron tab (directly or indirectly). That means one invocation updates all webs' WebStatistics.

Consolidating stats of sites in a federation

If you run a federation of sites, you should want to consolidate access figures of the master web and the slave webs.

As a part of content mirroring, access log files are copied from other sites in the federation so that consolidated stats can be compiled.

Let's assume the following federation of sites, which is borrowed from RepositoryForSiteAndWebMetadata#Site_metadata_and_web_metadata_f.

Name	Server	DataDir	PubDir	ViewURL	ScriptURL	ScriptSuffix
am	strawman	/disk0/data	/disk0/pub	http://strawman	http://strawman/cgi-bin
eu	woodenman	/d/twiki/data	/d/twiki/pub	http://woodenman/cgi-bin/view	http://woodenman/cgi-bin
as	tinman	/twiki/data	/twiki/pub	http://tinman/twiki	http://tinman/twiki/cgi-bin	pl

On the strawman server, let's assume that access log files are at /disk0/data/logs-as/logYYYYMM.txt. For log consolidation, other servers' logs would be stored in the following directories.

For Server	Path
woodenman	/disk0/data/logs-eu
tinman	/disk0/data/logs-as

Let's assume the site federation shown above having the following webs, which are again borrowed from RepositoryForSiteAndWebMetadata#Site_metadata_and_web_metadata_f.

Name	Admin	Master
WebOne	GodelGroup	am
WebTwo	EscherGroup	eu
WebThree	BachGroup	as
Sandbox	TWikiAdminGroup
Trash	TWikiAdminGroup

WebOne.WebStatistics is supposed to be of accesses on the servers strawman, woodenman, and tinman combined. And the WebOne.WebStatistics topic on all the sites in the federation would show the the same consolidated statistics.

WebOne.WebStatistics being a topic on the WebOne, as long as the consolidated stats is made on strawman (the master site) and written to WebOne.WebStatistics, the other sites get the same statistics when they mirror WebOne.

Read-only and slave webs

This is related to stat consolidation. Writing to WebStatistics needs to be suppressed on read-only and slave webs because you cannot. The save and upload scripts fail on a read-only or slave web.

%SYSTEMWEB% may be read-only but you may want to see how many page views it has. For that, you need a mechanism to write statistics to a different web.

Just like read-write webs, you need to have consolidated stats of read-only webs. And the same stats should be shown on all sites in the federation.

Users to be ignored

You may want to exclude search engine crawlers and other users from the statistics.

To cope with millions of page views and thousands of webs

This is not about ReadOnlyAndMirrorWebs per se, but a site employing it is likely to be large.

• If a site has millions of page views, it's not practical to run the statistics script from browser - it only times out.
• If there are thousands of web, it doesn't mean much to display links to WebStatistics topics of 50 webs in the alphabetical order at the bottom of Main.SiteStatistics.

Description and Documentation

Additional configuration parameters

The following configuration parameters will be introduced.

$TWiki::cfg{Stats}{DisableInvocationFromBrowser} = 1;
$TWiki::cfg{Stats}{IgnoredUsers} = [qw(root)];
$TWiki::cfg{Stats}{ReadOnlyWebs} = [qw(TWiki)];
$TWiki::cfg{Stats]{ReadOnlyWebStatsOn} = $TWiki::cfg{UsersWebName};
$TWiki::cfg{Stats}{LogFileGlob} = "/var/twiki/logs*/log%DATE%.*.txt";

{Stats}{DisableInvocationFromBrowser}

If it's true, when the statistics script is invoked from browser, it tells that "cannot invoke from browser". This is to prevent browser time out to happen on a site having millions of page views per month.

{Stats}{IgnoredUsers}

Can have a list reference having cUIDs of users to be ignored.

{Stats}{ReadOnlyWebs}

You can specify a list reference having names of read-only webs to have their statistics generated.

{Stats]{ReadOnlyWebStatsOn}

Specifies the web on which statistics of read-only webs are stored. The topic names are in the WEBNAMEWebStatistics format. e.g. TWikiWebStatistics

On read-only webs, you may have WebPreferences having:

%INCLUDE{"%USERSWEB%/%WEB%WebStatistics" disablefixlinks="on"}%

If you do this, users don't have to do anything special to see the statistics of a read-only web.

{Stats}{LogFileGlob}

If not defined, a single log file specified by {LogFileName} is read without doing anything special. If {Stats}{LogFileGlob} is defined, files matching the glob are read instead of the file specified by {LogFileName}.

They are read in parallel so that log entries are read in the chronological order instead of they are read file by file.

This is to make consolidated statistics of a federation of TWiki sites, which requires reading from multiple log files. Even if you don't have a federation, you may have multiple servers for a single TWiki site. And each server may have its own log file. e.g.

/nfs/twiki/logs/log201211.mars.txt
/nfs/twiki/logs/log201211.venus.txt

In that case, {Stats}{LogFileGlob} would have /nfs/twiki/logs/log%DATE%.*.txt.

Skip read-only and slave webs

Read-only webs and slave webs must not get WebStatistics topic update because TWiki::Store::saveTopic() fails on those webs.

To suppress links to WebStatistics of other webs at the bottom of WebStatistics

At the bottom of WebStatistics of all webs and %USERSWEB%.SiteStatistics, the list of WebStatistics of at most 50 webs are shown. If there are thousands of webs on the TWiki site, it does not mean a lot to show links to WebStatics of only 50 other webs.

If {NoInAllPublicWebs} is "on", the list is suppressed.

To this end, TWiki.WebStatisticsFooter is introduced and is included at the bottom of WebStatistics.

Site statistics

The site statistics is written on the {UsersWebName} web (Main by default), which might be a slave or read-only. It's processed and written only if the {UsersWebName} web is writable.

Examples

Impact

Implementation

-- Contributors: HideyoImazu - 2012-10-31

Discussion

Overall useful enhancements. A few comments:

• On read-only & disregarding mirror webs, why not simply ignoring the access control by the statistics script? I find it complicates things too much to generate the statistics in a different web (can be confusing to users and new admins). KISS.
• On {Stats}{DisableInvocationFromBrowser}, is the idea to use an IF in the stats page to hide the link? That is likely the easiest to implement.
• On {Stats}{NoWebStatLinksOnSiteStatistics}, it might be easier to %INCLUDE{ "%SYSTEMWEB%.WebStatistics" section="weblinks" } in all WebPreferences, and define a section in TWiki.WebStatistics that lists the webs, restricted to 32 (or so) webs, with a "more..." link if more than 32. With this there is no need to introduce a {Stats}{NoWebStatLinksOnSiteStatistics} flag. KISS.
• I don't understand = {Stats}{LogFileGlob} =

-- PeterThoeny - 2012-11-05

• I wasn't clear enough. So I wrote more at #Consolidating_stats_of_sites_in. I hope you understand why I'd like to do those things with read-only and mirror webs.
• Since running statistics takes a good amount of resources on a site having millions of page view, and there are people manually specifying URLs, I'd like to have a mechanism to forbid invocation from browser.
• I'm not sure I understand you. Are you saying changing all existing WebPreferences (thousands of them for me) is KISS? Changing Main.SiteStatistics at the bottom as follows is not so KISS but still it's relatively a small change.

%IF{"{Stats}{NoWebStatLinksOnSiteStatistics}" else="__Web Statistics Topics:__" %%IF{"{Stats}{NoWebStatLinksOnSiteStatistics}" then="HIDE{" else="SEARCH{" "WebStatistics" scope="topic" web="all" topic="WebStatistics" nonoise="on" format="[[$web.$topic][$web]]" separator=", " limit="100" }%

• I wrote more at #Consolidating_stats_of_sites_in and #Stats_LogFileGlob. I hope you understand what I'd like to by {Stats}{LogFileGlob}.

-- HideyoImazu - 2012-11-05

Ah, I was confused by "glob" as in "soft thick lump or mass" - May be better to name it {Stats}{LogFileGlobal} or {Stats}{GlobalLogFile}.

On {Stats}{NoWebStatLinksOnSiteStatistics}, a simple %INCLUDE{ "%SYSTEMWEB%.WebStatistics" section="weblinks" }% is shorter and more flexible (for mods) than the verbose %IF{"{Stats}{NoWebStatLinksOnSiteStatistics}" else="..."}%. That way you control what is shown in one place. BTW, you should not remove the "Related" links alltogether because of the SiteStatistics link.

-- PeterThoeny - 2012-11-05

I am coming back to the WebClusters feature we discussed in NoInAllPublicWebsOption. The {Stats}{NoWebStatLinksOnSiteStatistics} could be avoided with WebClusters because if a cluster is set in a web, the related links could show statistics links of its cluster. Food for thought.

-- PeterThoeny - 2012-11-05

I tend to think the word "glob" is established given http://en.wikipedia.org/wiki/Glob_(programming) and the File::Glob module bundled with the current perl distribution. I don't mind putting some comment about "glob" wherever appropriate, but {Stats}{LogFileGlob} seems brief and clear. The word "global" is probably where the term "glob" comes from, but "global" is much broader and less descriptive than "glob".

Regarding {Stats}{NoWebStatLinksOnSiteStatistics}, are we on the same page? It is about Main.SitePreferences, the site-wide statistics. It's not about WebStatistics of individual webs. And I'm talking only about the "Web Statics Topics:" section of the topic. I'm not talking about the "Related:" section.

-- HideyoImazu - 2012-11-06

I am OK with "glob".

On links, please note that all WebStatistics pages have related links at the bottom. My comments are related to that.

-- PeterThoeny - 2012-11-06

Ah. Sorry, I didn't know that %SEARCH{"WebStatistics" web="all, -%WEB%"}% is at the bottom of each WebStatostocs.

Now I think instead of introducing {Stats}{NoWebStatLinksOnSiteStatistics}, referring to %NOINALLPUBLICWEBS% seems better for consistency. If you set %NOINALLPUBLICWEBS% to avoid "In All Public Webs" option on the "More topic actions" page, most likely, you want to avoid seeing WebStatistics of all webs. I retracted {Stats}{NoWebStatLinksOnSiteStatistics} and introduced #To_avoid_thousands_of_links_to_W

Rather than putting %INCLUDE{ "%SYSTEMWEB%.WebStatistics" section="weblinks" }% at the bottom of WebStatistics, how about this: the lines from "__Noets:__" till the end are replaced by %INCLUDE{"%SYSTEMWEB%.WebStatisticsFooter"}% ? This is similar to each WebPreferences has %INCLUDE{"%SYSTEMWEB%.WebPreferencesHelp"}%.

-- HideyoImazu - 2012-11-06

Sounds good on including a WebStatisticsFooter.

-- PeterThoeny - 2012-11-07

Accepted by consensus reached at JerusalemReleaseMeeting2012x11x09.

-- PeterThoeny - 2012-11-09

Now that NoInAllPublicWebsOption uses {NoInAllPublicWebs} instead of %NOINALLPUBLICWEBS%, the To avoid thousands of link section has been changed accordingly.

-- HideyoImazu - 2012-11-09

Site statistics section has been added.

-- HideyoImazu - 2012-12-12