Please help to refactor these ideas into ConsolidateNotification

If the conversations below repeat what's in ConsolidateNotification, please please help by deleting them.

---

---

Ok, let's start a new effort. First let's agree on the specification on what to code, and then try to implement it. The goal of this page is to try to mature a solution to be implemented later by any takers, me by default if nobody else wants it, but this is not "official" as I am not part of the CoreTeam. Feel free to contribute). I do not plan to code this before May 2003

Let the user choose:

1. Which topics to monitor
2. At which interval
3. What to receive
4. How to receive it (e.g. do we try Yahoo Instant Messenger first and, if not logged on, try Jabber then email.) Also included could be a personalised RSS feed. -- MRJC.
   • CN: we have two options for this: use a separate option such as Notify Medium, or encode this in the "email", like (syntax to be defined): Email: IRC:Colas@irc.ilog.fr (my preferred solution, I think, as this functionality can be thus added to all parts of TWiki handling sending email)

Please comment, add your own proposals or correct/comment mine (especially naming of options, as I am not a native speaker):

I have put question marks on features that I do not feel like implementing in a first version (help welcome with them).

Colas original proposal: Colas1

In WebNotify, entries will be of the form:

• Users's_Wiki_Name [Option_Name: Option value]

The options being either on the same line, or continuation lines, or sub-bullets. Defaults options can be set in the user Home page as:

• Option_Name: Option value

The Options being:

• Email (note that this way, the Email field of the home page acts as a default option). Rationale: being able to send some notifications to other addresses.
  Option: the email could be an InstantNotification "address" (syntax to be defined)
• Notify Interval value being a number of hours, between 1 and 24. Rationale: basically people want to be notified nearly immediately or once a day, or never. I think that a minimum of one hour promote less "kneejerk" replies, but this could be added () by adding a possible value "immediate" or "0" in a further version. Thus there is no need to a more fine-grained interval.
• Include a regexp of topics that will be checked for changes. Rationale: the most asked for feature.
• Include Body a regexp of topics that will be checked for changes if their contents matches. Rationale: people are beginning to use Wiki for some database-like apps, and want to be notified on field values (e.g: module name for bug reports, where topic names are generated)
• Exclude a regexp of topics that will be excluded from changes
• Exclude Body a regexp of topics that will not be checked for changes if their contents matches
• Notify Diffs if on will send the diffs also in the mail. Rationale: also asked for, but may not needed. In any case the number of changes and a direct link to the diffs should be sent in the (html) mail.
• Notify Textual if on email will be sent as pure ascii, no HTML.
• Notify Attachements if on will notify if an attachement was done/updated with not edit.

Include and Exclude option can be repeated and are cumulative. If the first one encountered is an Exclude, a Include: .* declaration is implied before.

Options can optionnally be separated by commas and dashes.

User name can be a group name, same rules apply.

Option naming: I have tried to use explicit naming, - "Notify Interval" rather than just "Interval" -, which should be clearer when including these options as default in the home topic. Alternative ways (please comment)

1. Use a terse version (Interval). People will know what they mean.
2. Use rather a TWiki syntax in the home page: Set NOTIFY_INTERVAL = 12 But if we force this syntax in the WebNotify topic, il will be cumbersome, and if not, the syntax will be different in Home and WebNotify, bad for the users.
3. Use the terse (Interval) version, but tolerate prepending "Notify " to all of them, for use in the Home page if wanted

Example:

• ColasNahaboo - Include: Koala, Notify Textual: on, Include Body: KoalaSkin

or, to track all topics with "Notify" in their name, except WebNotify, twice a day:

• ColasNahaboo
  • Include: Notify, Exclude: WebNotify
  • Notify Interval: 12

Planned Implementation

Each hour, a scheduled script will scan all the WebNotify topics, building a set of options per web per users from entries relevant for this run (wrt "Notify Interval"). It is relevant if current_hour modulo notify_interval is 0. It will remove the ones already sent to the user in the relevant timeframe (the notify_interval of the entry) as seen on the sent_log It will then compute (per user) the set of topics that should be notified, and send them and put them in the sent_log. (at 0h00 run, sent_log and the changes log are cleaned)

Observations: notify_interval can only take the actual values 1,2,3,4,6,12,24 The system could allow for bigger interval. Smaller intervals may not be realistic, depending of the time taken by the script to perform its tasks.

Optional: track WebNotify changes with the on-save handler, and compiles what to do in some way. This could detect immediate notification demands, and set up the needed triggers.

I will try to code this as as standalone replacement to mailnotify + some patches.

-- ColasNahaboo - 08 Apr 2003

Great work for taking the initiative on this, Colas. You have my thanks.

As for the interval: I think we should take advantage of the fact that there is a difference between TWiki collecting events and notifying of them. You could collect the events into a topic using the on-save handler hook mechanism that was pioneered in ImmediateNotifyPlugin. On your periodic basis you could sweep through this table of events looking who to notify. This has the advantage of unifying the process for those who want periodic and instant gratificationnotification.

Note that ImmediateNotifyPlugin does not handle an on-attach handler so if one attaches but does not also edit no notification is generated.

-- MartinCleaver (MRJC) - 08 Apr 2003

Actually I was planning to use the existing data/XXX/.changes files to store/retrieve changes.

As for Attachements, good remark. I have added the Notify Attachements option.

PS: I am not a fast coder, I'd rather wait, toying with the specs till they feel to "do the right thing", then code And I may decide not to implement everything.

-- ColasNahaboo - 09 Apr 2003

One thing that this effort could definitely use IMHO is a predefined "notification method" API similar to the one I have set up in my ImmediateNotifyPlugin. That would allow the TWiki administrator to decide which methods to allow, and for developers to write new notification methods in much the same way they write TWiki plugins now.

-- WalterMundt - 09 Apr 2003

Yes. I was even thinking of "delegating" (I do not know how yet) Immediate Notification to your plugin... But it may be simpler to just use your API...

-- ColasNahaboo - 09 Apr 2003

Why not store the notification preferences in the user's profile page and stop reading the WebNotify page? TWiki's mailnotify script already handles placing the email address in the profile page, why not everything else? Shouldn't a goal for TWiki be that everything about a user is stored in that user's profile?

If there is a still a need for the WebNotify page, it could be come a dynamically created page via %SEARCH after the preferences are all stored with the user's profile.

-- TomKagan - 09 Apr 2003

Now, that is quite interesting. It will bring:

• easy implementation of buttons "email me on topic change", as these automated procedures will have only the home page to edit, rather than a common WebNotify topic where conflicting edits can occur between different people
• a simple way to see all our subscriptions
• a possibility to specificy regexps also on webs (For instance .*\..*Koala.* to monitor all topics with Koala in their names in all the webs.

But, we will have to find how to specify "contextes": e.g. Notify interval 12 hours for one web, 1 hour for another.

I have not much time now, I will try to see how we can design the specs here. The more I think of it, the more it seems as a good idea.

-- ColasNahaboo - 09 Apr 2003

Alternatively, you could stick subscription information in the meta data for the topic. In some ways it is more efficient to have it on the topic because TWiki will already have the topic open when processing the notification.

• (This conversation rests on the fact that TWiki has no relational database: the Topic-Subscriber would be best satisfied by a co-dependent join because this would ensure both topic and author would survive a delete or rename. But I digress...)

In any case, the UI of each page requires a 'Subscribe me' and 'Unsubscribe me' button.

Advanced consideration: some people might possibly prefer different types of notification depending on the topic.

-- MartinCleaver - 10 Apr 2003

---

Colas "Home Page" proposal: Colas2

Modifying the topic can be a bit problematic for "meta" modifications, as the topic can be locked for editing by another user.

So, Email notifications would be stored in the User TWiki Home Page, (no more WebNotify), of Users or Groups in the form of

• bullet list items beginning with Email Notify: (or Notify On, Monitor, Email On Changes, ...)
• followed by: what to monitor (regexp matching on Webname/Topicname).
  • I use Webname/Topicname rather than Webname.Topicname as . (dot) is awkward to match with regexpes, suggestions?

Followed by options either on the same line, or continuation lines, or sub-bullets, in the form: Option Name: Option value . The Options being:

• Email (note that this way, the Email field of the home page acts as a default option). Rationale: being able to send some notifications to other addresses.

Option: the email could be an InstantNotification "address" (syntax to be defined)

• Interval value being a number of hours, between 0 and 24. (or more if we know how to implement it) Rationale: basically people want to be notified nearly immediately or once a day, or never.
  0 meanin immediately, and being implemented via the ImmediateNotifyPlugin by WalterMundt
• Include a regexp of topics that will be checked for changes. Rationale: the most asked for feature.
• Include Body a regexp of topics that will be checked for changes if their contents matches. Rationale: people are beginning to use Wiki for some database-like apps, and want to be notified on field values (e.g: module name for bug reports, where topic names are generated)
• Exclude a regexp of topics that will be excluded from changes
• Exclude Body a regexp of topics that will not be checked for changes if their contents matches
• Diffs if on will send the diffs also in the mail. Rationale: also asked for, but may not needed. In any case the number of changes and a direct link to the diffs should be sent in the (html) mail.
• Textual if on email will be sent as pure ascii, no HTML.
• Attachements if on will notify if an attachement was done/updated with not edit.
• Author regexp of author names; this is not terribly interesting, but a prerequisite for the next one:
• Group regexp/list of group names; matches, if the author is member of such a group, e.g. TWikiAdminGroup -- PK
  CN: maybe redundant with Author... we could just provide this functionality if you specified a group name in the Author field
  PK: yes, you could do this by listing authors. But then, every subscriber has to duplicate + track the information from TWikiGroups.
• Form list of form names; this is redundant to include regexps, but readability should be worth it, e.g. * Form: BugReports, FeatureEnhancementRequests -- PK
• RSS write given number of notification records to static RSS XML, maybe sth like /twiki/rss/UserName.xml -- PK

Include and Exclude option can be repeated and are cumulative.

Options can optionnally be separated by commas and dashes.

A script should be provided for putting buttons "Monitor this web" and "Monitor this topic" to edit automatically the Home Page (append to end, better would be to append to the block of current email declarations)

Examples:

In my Home Page I wuld write this to be notify obn changes of topics in web Koala whose contents contains the string KoalaSkin

• Email Notify: Koala/, Textual: on, Include Body: KoalaSkin

And, to track all topics with "Notify" in their name, except WebNotify, in all webs, twice a day:

• Email Notify: .*/.*Notify
  • Exclude: WebNotify
  • Interval: 12

You can add as many "* Email Notify:" item entries to your home page, each being independently processed (for interval, etc...)

-- ColasNahaboo - 14 Apr 2003

Putting all notifications into user pages instead of meta data offers more advantages:

• no complicated and slow %SEARCH to review all your subscriptions
• if you want to add a list of n page names, you need 1, not n edits
• likewise, the page revision history is not cluttered with subscriptions
• suppress notifications about notifications being added by not subscribing to the Main web
• if you want to specify groups of pages with one expression, you need anyway a place outside of the affected pages.
• as I like the idea of having decently readable keywords instead of %FOO %BAR stuff very much, this must be confined to user pages. Otherwise, ambiguities might become a problem.

Sequential scope: Peter1

Ad different notification policy per web or topics: Why not simply allow multiple occurences and scope sequentially? Like so:

• Notify Interval: 24h
• Include: .*notify.*
• Include: .*this.*
• Notify Interval: 1h
• Include: .*that.*
• Include: NewEmailNotificationSystem

For the second include being more specific than the first, the second interval should apply. I.e., for every matching include/exlude, the last common definition of interval, type etc. should apply. This probably means, that the most typical settings should be the last section, where automagic subscriptions will be appended.

Ad wishlist: Above, I added Author, Group, Form, RSS to the keyword wishlist.

-- PeterKlausner - 15 Apr 2003, clarifications 22 Apr 2003

Ah, in my mind, each definition is local only to its bullet item. Your example would be thus written:

• Email Notify: .*notify.*
  • Interval: 24h
• Email Notify: NewEmailNotificationSystem, Interval: 1h

The two forms of specifying Interval being equivalent (continuation line or sub-item, or on the same line). Each 1st-level entry being independent.

(I added also some remarks on the Group: field proposal)

-- ColasNahaboo - 20 Apr 2003

Re: the two different definitions: I would vote for the one that requires the least typing by the user, which I think is the first one. (If the computer has to work harder, let it do so (unless it significantly increases the response time.))

-- RandyKramer - 20 Apr 2003

I have added "IDs" to propositions to be able to tell them apart (Colas1, Colas2, Peter1...). Randy, which one do you mean by "the first one" ?

On goal to reach: I first that we should aim at the more obvious/understandable/maintainable for the user, not necessarily what requires the less typing (in many cases people do copy/paste rather than typing). But then... I do not like perl

On letting the computer do the work: I agree, but what the computer does must still be understandable by the user, not some black magic (this is a general remark; I do not know exactly what you were meaning).

-- ColasNahaboo - 21 Apr 2003

Sorry, I wasn't very clear! I was comparing the following two choices:

option 1:

• Notify Interval: 24h
• Include: .*notify.*
• Notify Interval: 1h
• Include: NewEmailNotificationSystem

option 2:

• Email Notify: .*notify.*
  • Interval: 24h
• Email Notify: NewEmailNotificationSystem, Interval: 1h

It looks like there would be less typing with the first option, as it looked like I could specify an interval and then list one or multiple pages under that interval. In the 2nd option it looks like I have to specify the interval for each page I list.

regards,
-- RandyKramer - 22 Apr 2003

Ok, I see your point. The "context-free" version of Option 2 is more open to use via tools (forms to edit the options, buttons on pages "monitor this page"). A way to get the best of both worlds would be to specify default properties "cascading" to following "Email Notify" declarations via another declaration, e.g: Email Notify Settings:, which gives:

option 3:

• Email Notify Settings: Interval: 24h, Exclude: Web.*
• Email Notify: .*notify.*
• Email Notify Settings: Interval: 1h
• Email Notify: NewEmailNotificationSystem

Now an open question would be: Does the Settings cascade to all the "Email Notify" beneath it, or only up to the next Settings? i.e., does the Exclude: Web.* setting applies also to the NewEmailNotificationSystem (option 3a ) or not? (option 3b )?

3a would allow to express things tersely, but may bring headaches via unseen cascadings (like what happens with X ressources or CSS style sheets), and would make automatic editing of notifications trickyier.

PS: I do not think I will be able to work on this before May 25.

-- ColasNahaboo - 13 May 2003

Colas, I see your point / understand your question. Not sure I have a strong opinion either way, as long as (for option 3A) there is a way to cancel the "Exclude: Web.*" with something like "Allow: All" or "Allow: Web.*"

-- RandyKramer - 16 May 2003

Well, if you look at it, Option 3's "Email Notify Settings:" plays this role of cancelling/resetting the previously specified settings...

-- ColasNahaboo - 16 May 2003

Great, thanks!

-- RandyKramer - 17 May 2003

No, guess what... I didnt have time to devote to this until now, but it is going to change as my new job position will make me officially in charge of TWiki on our intranet, so I definitely will begin to code this... as soon as I come back from vacations,on July 21

In the meantime please review the design and provide feedback!

-- ColasNahaboo - 24 Jun 2003

Some Use Cases/User Stories

Some points:

• Email notification is currently relatively hard for people to grasp who haven't used a Wiki of any kind at all before
• The discussion above is great as a configuration approach, but IMHO sucks as a user interface.

Some use cases/user stories:

• The user wishes updates to a web, they click on the "subscribe to this web" button, enter their username & password and go away happy.
• The user wishes updates to a topic/particular discussion, they click on the "subscribe to this topic" button, enter their username & password and go away happy.
• The user wants to unsubscribe from a topic or web. They go to their home page, click on "subscriptions", and delete the topics/webs they're no longer interested in. (From the webform at the bottom of the page) ... and go away happy.
• The user wishes to customise their notification - they go to their home page, click on "subscriptions", and click & check things that describe regularity of update, style of update (IM, email, topic list, diffs) and go away happy.

Subscription really should be just a matter of click, username/password 'n done.

-- MichaelSparks - 24 Jun 2003

A "click and you're done" (subscribe to this wiki, subscribe to this web, subscribe to this topic, subscribe to this category) is something that would be best done in a plugin.

BTW implementors, I did quite a lot of work to make sure that the "meaty" bits of the action notifier scripts were kept out of bin. Can you do the same here, keep the interesting bits out of bin please? That way we can re-use them in other places - such as the action tracker V3.

-- CrawfordCurrie - 25 Jun 2003

To Crawford: yes, I agree. I tend to put things in bin/ for bad reasons, namely that since patches to the core get so many years to be accepted, a simple script to drop in bin/ is the easy way for people to try this change, without fear of breaking anything.

Another point, more as a reminder to take into consideration: I saw some users edit their wiki home page to change their Email: "field" for aesthetics (putting the word Email in bold), this would break scripts expecting to find the Email field. So I think we should modify twiki to be able to get also the mail in the form:

   * Set EMAIL = foo@bar.com

As this syntax should be more obvious for users that this field is used by the system

-- ColasNahaboo - 08 Aug 2003