Feature Proposal: Real-time notification by MailerContrib

Motivation

There is a need for real-time email notification of changes.

Description and Documentation

MailContrib will have afterSaveHandler() and become able to send real-time email notification. WebNotify is read to determine who is notified with what topic.

To keep the current behavior, real-time notification is turned off by default. If the MAILERCONTRIB_RTNOTIFY preferences variable is set to 'on', the feature becomes active.

A real-time notification email is sent in the similar manner as a change summary sent by tools/mailnotify.

1. WebNotify is read and the users who get a notification are determined.
2. The rtmailnotify template (instead of mailnotify for tools/mailnotify)is read and an email is composed accordingly.

When a person hear "change notification", they tend to think of it a real-time notification rather than once a day change summary. With this feature, most users would use real-time notification only. For that, MAILERCONTRIB_NOTIFY is set to 'off'.

Some users may be fed up with frequent real-time notifications and switch to once a day change summary. This is achieved by setting MAILERCONTRIB_RTNOTIFY off and MAILERCONTRIB_NOTIFY on.

Few users would use both real-time and batch notifications.

Types of updates

To fine-tune notification behavior, you need to be aware that there are following types of update actions.

With topics
Action	Event name
New topic creation	new
Topic update	updt
Topic move/rename	move, recv (if a topic is moved to a different web)
Topic deletion	del
With attachments
Action	Event name
New attachment upload	newa
Attachment update	updta
Attachment move	movea, recva (if an attachment is moved to a different web)
Attachment deletion	dela

When an update is made, the corresponding event is fired. When a topic or an attachment is moved to a different web, as shown on the table above, two events are fired - one event on the source web and another event on the destination web.

You can differentiate notification actions depending on an event fired. You can also cause notification for some actions while suppressing notification for others.

When an update event is processed, TWiki enters the context whose name is notify_ followed by the event name. For example, when a new topic is created, TWiki enters the context notify_new.

While tools/mailnotify is being executed, TWiki is in the mailnotify context.

WebNotify

Since each line on WebNotify has its variables expanded, you can have the following line on WebNotify so that the user JoeSchmoe is notified only with new topic creation.

   * %IF{"context notify_new" then="JoeSchmoe"}% 

WebNotify format would be enhanced from:

three spaces * subscriber [ : topics ]

to:

three spaces * subscribers [ : topics ]

where subscribers are comma separated list of wiki names or email addresses. This provides an opportunity to specify subscribers dynamically as follows.

   * %SOMETHING_YIELDING_SUBSCRIBER_LIST{param="value"}%

Real-time notification template

tools/mailnotify uses the mailnotify template to compose a notification email. Real-time notification being significantly different from tools/mailnotify, it uses a different template rtmailnotify.

Preference variables

Preference variable	Description	Default
MAILERCONTRIB_NOTIFY	If it's 'off', tools/mailnotify doesn't generate notification email for the web	on
MAILERCONTRIB_RTNOTIFY	If it's 'on', real-time notification is sent for the web	off
MAILERCONTRIB_NOTIFYTMPL	The email template for tools/mailnotify	mailnotify
MAILERCONTRIB_RTNOTIFYTMPL	The email template for real-time notification	rtmailnotify
MAILERCONTRIB_EXCLUDEUPDATER	If it's 'on', the updater herself is excluded from real-time notification	off

Subweb may use WebNotify of its parent

If you have a number of subwebs, you may want to configure notification at a single location - the top-level web. If a subweb doesn't have WebNotify, it's closest ancestor's WebNotify is used.

Why not implement a plug-in separated from MailerContrib?

It's possible to implement a plug-in for real-time notification separated from MailerContrib. Actually, I did that but the result wasn't satisfactory for the following reasons.

• You need a new topic for real-time notification spec, e.g. WebRTNotify, which is similar to WebNotify and confusing.
• Switching between real-time notification and batch notification is cumbersome.

Examples

Impact

Implementation

-- Contributors: HideyoImazu - 2012-11-20

Discussion

Summary of discussion at JerusalemReleaseMeeting2012x11x22:

• Needs more design work.
• PTh goal is to simplify UI so that non-technical users can use it. Initially suggested at WatchlistPluginDev.
• PTh suggests:
  • Just one "watch this topic" menu item / button
  • A watch page to watch all page changes of interests, like in mediawiki
  • In addition, the user has a picklist to get notified by email in real-time, in batch mode, or not get notified at all by email
  • Possible to watch whole web: The watch menu item in WebHome could be labeled "Watch the Foo web", for all other topics in that web it could be "Watch this topic". Or the "Watch the Foo web" menu item could be an additional item to the "Watch this topic" item in WebHome.

-- PeterThoeny - 2012-11-22

I understand the enhancement proposed here doesn't get in the way to the final goal of easily subscribing to individual topics in a few clicks. And it lays a good foundation for real-time email notification. What do you think about implementing this enhancement as a step toward the final goal?

In addition to notification of individual topics and and entire web, there is a need for notification of a set of topics. I made a bulletin board application on TWiki which sends a new entry and entry update by email in real-time based on a real-time notification feature I experimentally implemented.

Notification specification by WebNotify is not for ordinary users. But it works OK for an entire web and a set of topics.

Meanwhile, subscription to individual topics is better to be stored on individual topics instead of a WebNotify like topic. A topic may be renamed, moved, and deleted. Subscription information stored on the topic doesn't need to be modified in those cases. For this ideal, we need to come up with a way to store subscription information in a topic. And then we need to think about user interface to manage the information.

-- HideyoImazu - 2012-12-26

I feel that the spec as stated is quite complex, too complex for most users to understand. How about implementing this feature as a new plugin?

While I agree that subscription should not be stored in WebNotify, I don't think that storing it in topics themselves is good, because it would show topic changes with each subscription/unsubscription. This is too noisy; I think topic should be mainly reserved for content and topic related meta data. I think it is better to store subscription data in plugin working area. The plugins API has a afterRenameHandler callback that can be used to keep plugin working data in sync with topics.

-- PeterThoeny - 2012-12-28

I updated the spec of the proposed WatchlistPlugin in WatchlistPluginDev.

-- PeterThoeny - 2012-12-28

Please be noted that the feature to differentiate types of update is an expert feature most users won't care about. Ordinary users won't be affected by the additional feature. But for some applications, it's handy.

There is a clear need for real-time notification. Some web owners want to know when updates are made on their web. A good number of users who want real-time notification don't want daily change summary for the same web. It's natural for a single notification extension to provide both real-time and once a day summary in a coherent manner based on WebNotify. Introducing something like WebRTNotify for real-time notification is confusing for users.

-- HideyoImazu - 2013-01-03

Does the spec of the proposed WatchlistPlugin as described in WatchlistPluginDev fit your needs?

-- PeterThoeny - 2013-01-03

No, this doesn't meet some requirements I have.

First of all, the main point of this proposal is to have real-time email notification. While WatchlistPluginDev doesn't seem to be about real-time notification. Am I correct?

I have a bulletin board implemented on TWiki, on which every time a new item is posted or an item is updated, a set of users need to be notified by email.

-- Hideyo Imazu - 2013-01-07

As proposed, the WatchlistPlugin it has real-time notification. Click on the "Preferences" tab of the mockup, it has an option for "Immediate: Get notified on each topic change on watch list."

-- Peter Thoeny - 2013-01-07

Understood. Still, I need a capability to specify the recipients of real-time notification dynamically to meet the bulletin board requirement. For that, enhancing MailerContrib and using WebNotify is easier.

-- Hideyo Imazu - 2013-01-08

We need both daily change summary and real-time change notification. For that we need both afterSaveHandler() (for real-time notification) and tools/mailnotify like command (for daily change summary) anyway. Meanwhile, I understand the problem WatchlistPluginDev is trying to solve is the user unfriendly-ness of WebNotify in subscribing/unsubscribing to webs and topics. That said, WatchlistPluginDev and this proposal can be combined. The notification feature (both real-time and summary) will see both WebNotify and the new subscription data managed by a more user-friendly user interface. How does this sound?

-- Hideyo Imazu - 2013-01-10

I was thinking more of keeping MailerContrib (with WebNotify topics) and WatchlistPlugin (with <WikiName>WatchList topics) separate, ship both in next release(s), and at some time retire MailerContrib.

On combining, not sure if it makes sense in terms of e-mails sent. The MailerContrib currently sends one e-mail per web, the WatchlistPlugin is likely to send one e-mail for all watched content.

-- Peter Thoeny - 2013-01-10

I forgot to mention WebNotify format extension planned, which is now described at #WebNotify. I have a bulletin board application on which a set of users are notified when a new article is posted or updated. This cannot be achieved by WatchlistPluginDev.

On a different note, my federated TWiki sites probably cannot use WatchlistPlugin because the Main web may not be writable depending on site - Main is mirror and updated only on the master site of the Main web.

-- Hideyo Imazu - 2013-01-11

Is the subscribers (list of users) necessary? Is that already covered by groups? A group can be specified as recipient.

It looks like your planned enhancements and the WatchlistPlugin have a bit different goals, so may be better to implement separately.

I leave it up to you if to create a new plugin or if to enhance the current MailerContrib. If you enhance the MailerContrib make sure to separate the docs into regular end user docs and admin user docs, with the latter hidden in a twisty.

-- Peter Thoeny - 2013-01-14

I know you can put a TWiki group as a recipient on WebNotify. That's not enough for the TWiki application I'm talking about. Users of the application have several attributes, which are put on a table on a topic. Those attributes are referred to from several places in the app. Real-time change notifications by email are sent depending on user attributes.

-- Hideyo Imazu - 2013-01-15

Now that the general direction is to switch to WatchlistPlugin from MailerContrib, putting a lot of effort to enhance MailerContrib doesn't make a lot of sense. As such, I abandon this feature proposal.

-- Hideyo Imazu - 2013-06-17

OK. You could consider enhancing the WatchlistPlugin. Look also into the newish NotifyAuthorsPlugin which is designed to be used in workflow apps.

-- Peter Thoeny - 2013-06-17