Enhanced TWiki Mail Notification

Please read ConsolidateNotification and MailerContrib

Intro

I have found many ideas on TWiki.org about mail notification enhancements but no code proposed. We really needed some new features on mail notification, so I tryed to developped them in respect of TWiki stability, trying to offer an anwser to the same kind of needd I could found in many over mail notify topics available in TWiki Codev.

An another important think was, in my opinion, to put mailnotify functions in a dedicated lib MailNotify.pm so to encapsulate these features, like it is nicely done in TWiki plugins.

Hope this little work will enjoy every twiki user and the TWiki Codev team too as I did it in the hope they will accept it in futures releases

New features

Per WEB topic exclusion

Topic WebNotifyExclusion list (TWiki list) all TOPICs that should not be notified when changed. WebNotify topic itself could be listed if you think that nobody should receive a mail when this topic is changed.

Content of Topic WebNotifyExclusion could be :

• WebNotify
• WebHome

etc...

Include or/and exclude TOPICs against topic names, topic content, topic author

you can define, for each user or group, in each WebNotify topic, a regex KEY to restrict TOPIC you accept to be notified for :

• PatrickNomblot IN{TOPIC="Notification"}
• TWikiAdminGroup IN{TEXT="I love PERL"}
• DreamTeamGroup IN{AUTHOR="Main.ColasNahaboo"}

To exclude some topics, use keyword NOT{} and define TOPIC, TEXT or AUTHOR matching values :

• PeterThoeny NOT{TOPIC="^(WebHome|WebNotify)$"}
• AndreaSterbini NOT{TEXT="I hate PERL"}
• JohnTalintyre NOT{AUTHOR="Main.PatrickNomblot"}

Regex is tested against the TOPIC name (TOPIC), TOPIC content (TEXT), last topic change author WIki username (AUTHOR). Case is important.

Examples

• RichardDonkin NOT{TOPIC="WebNotify|WebHome" TEXT="I hate PERL" AUTHOR="Main.WalterMundt"}

Richard will be not notified if a TOPIC name is like WebNotify or WebHome, and also not notified for any topic containing text I hate PERL and will never be notified about TOPIC changes if Walter is the last author at the time of notification.

• PeterThoeny IN{TEXT="Main.PeterThoeny"} NOT{AUTHOR="Main.PeterThoeny"}

Peter will be notified only for all topics containing text PeterThoeny but not for those for which he is the author at the time of notification.

per user mailnotify template selection

Just add TEMPLATE="template_name" in IN or NOT command to select a specific mail template. This afect only current user or group. Mail notify templates are stored in templates dir, must be named as mailnotify___blablabla__.tmpl .

example :

• PatrickNomblot IN{TOPIC="Notification" TEMPLATE="Simple"} will select template file mailnotify_Simple.tmpl
• TWikiAdminGroup IN{TEXT="I love PERL" TEMPLATE="SimpleHtml"} will select template file mailnotify_SimpleHtml.tmpl
• PeterThoeny NOT{TOPIC="^(WebHome|WebNotify)$" TEMPLATE="NiceHtml"} will select template file mailnotify_Nice.tmpl

Keep a copy of sended mails

Option -k make possible to keep mail in .mail files, in dir data/. This help debugging or track sended mail. One .mail file is created per mail adress.

Do not send mail, just show what notify would do

Nice for debugging, mail is not sent but stored in .mail files. Notify time is unchanged so you can repeat notify mechanism as many time as you need.

Find TWiki users email address in LDAP (if LDAP defined)

If you have an LDAP server and would like to get adresses from it rather than from TWiki, just set LDAP server adress and LDAP BASE in TWiki.cfg file.

List of TWiki files changed :

• bin/mailnotify --> only main, functions moved into MailNotify.pm lib, in PERL strict mode, also corrected some wrong links in mail produced by original script
• lib/TWiki.cfg --> added LDAP server and base optionnal values
• lib/TWiki.pm --> function specific to mail notification moved into MailNotify.pm lib.
• lib/MailNotify.pm --> the mailnotify core.
• templates/mailnotify.tmpl --> added warnings, correct some details

Files are the one we currently use, you should merge changes to your files if you also made some changes from TWiki20030201 release.

per web mailnotify frequency

mailnotify bin allow to specify web names, space separated, in inline command. This allow to run per web, if needed, each notification session. This feature as also be done for debug purpose, or to be abble to run it on hidden webs too.

Possible enhancements

• per user mailnotify frequency
• convert , to | as for search regex
• be able to specify case significant or not
• define notification time interval (easy to do (now) but I don't need it for now, so... )
• show Topic list if plugin syntax used : %IN{...}% %NOT{...}%
• etc...

How to install

• copy mailnotify in bin/mailnotify
• copy MailNotify.pm in lib/MailNotify.pm
• copy mailnotify.tmpl in templates TWiki dir
• copy mailnotify_simple.tmpl in templates TWiki dir (optional)

If you use LDAP, you also need to make these changes :

• apply TWiki.cfg patch :
  • new LDAP vars added
• apply TWiki.pm patch :
  • new LDAP vars added,
  • force LDAP login in lower case as login is not case significant for LDAp but was in TWiki,
  • delete sub getEmailNotifyList which is moved into MailNotify.pm

Note : LDAP perl modules are optionnal. Thank's to KeithHelfrich for that enhancement (18 Mar 2004)

I hope Codev team will like this, I could make some enhancements or changes if needed but would appreciate that future TWiki releases offers this enhanced TWiki Mail Notify.

-- PatrickNomblot - 26 Sep 2003

This looks very interesting - I did some work on this code a while back but you've taken it much further. Definitely a good idea to define a MailNotify.pm module, though it should probably be defined as TWiki::MailNotify like other modules.

Perhaps you could produce patches based on PatchGuidelines, particularly for TWiki.cfg and TWiki.pm, to make it easier for people to try out your code?

-- RichardDonkin - 07 Oct 2003

And a demo would be nice

-- MartinCleaver - 07 Oct 2003

Patrick, your adaptations are very interesting.

Three questions or requests:

1. Is it possible to by default exclude notification of changes made by onesself? I consider TWiki telling me things that I already know (like that I made a change) to be spam.
2. Is it possible to request that the notifications get rolled into one? I sometimes get 16 emails on every run of the script
3. Did you implement "mailnotify -q" to silence the script when it is run by cron?

Lastly I wonder whether it would be better to have a control panel launchable from the person's home page. At the very least we need to include a search for the user occuring on all the webnotify pages. (Hmmm... This leads into the notion that mailnotify should be an event that is serviced by plugins, and that plugins should have wizards triggered by a user visiting their home page.)

-- MartinCleaver - 09 Nov 2003

1. Yes, it is an easy feature to exclude notification of changes made by oneself. This is also possible via option NOT{AUTHOR="Main.PatrickNomblot"} but this feature could also be activated via a global option.
2. notifications rolled into one is also a nice idea but maybe need to prepare mails on disk, append content per web, and send at the end... why not
3. ooops, I think there is a bug for the "mailnotify -q" , debug is set to 1 at line 45, it should be 0 to be quiet quiet.

A control panel could be nice... here is a search of web you are registered on. In my case, I have changed all WebNotify Topics to include a TOPIC header and a TOPIC Footer, thus it is easy to change infos and to do a search on a unique key which is the include itself. Here is what you get but you need to be logged in:

Web Home	Notify Topic
Blog	WebNotify

Codev	WebNotify
Codev	MailNotificationEnhanced
Codev	MyLastEditedTopics

Main

WebNotify

Plugins

WebNotify

Sandbox	WebNotify
Sandbox	RobertDeBruinSandbox

Support	WebNotify
Support	SID-00886
Support	SID-00659
Support	SID-00458
Support	WebNotifyNotEmailing
Support	MailNotifyCrashes
Support	MailNotifySendsAllOrNoChanges
Support	MailerContribFailedToCreateFile

TWiki

WebNotify

TWiki02

WebNotify

TWiki03

WebNotify

TWiki04

WebNotify

TWiki04x01

WebNotify

TWiki04x02

WebNotify

TWiki04x03

WebNotify

TWiki05x00

WebNotify

TWiki05x01

WebNotify

TWiki06x00

WebNotify

TWiki06x01

WebNotify

WikiWed

WebNotify

-- PatrickNomblot - 12 Nov 2003

• Excluding your own changes by default is necessary. I think there would be few people that would need it on.
• I get 16 mails everytime it is run - if you can roll them into one I would appreciate it.
• Is there a mode to not only send the topic names and short summary but the change in the topics as well? Is that what the templates provide?
• IMO all templates should be topics in a web. Can you make this an option as well?
• My users definitely have differing time interval requirements
• Although I don't mind the syntax I am sure that many of my users would balk at it. Perhaps the subscription should be in webnotify but the options on the person's home page. I'm not sure.

Cheers. PS. Any chance you could answer my survey? Thanks.

-- MartinCleaver - 14 Nov 2003

Patrick,

Why in TWiki.pm do you write:

+     $webNameRegex = qr/[$upperAlpha]+[_$lowerAlphaNum]*/;

Are you also running WebNameAsWikiName?

 1028  cd patches/
 1029  wget https://twiki.org/p/pub/Codev/MailNotificationEnhanced/mailnotify
 1030  wget https://twiki.org/p/pub/Codev/MailNotificationEnhanced/MailNotify.pm
 1031  wget https://twiki.org/p/pub/Codev/MailNotificationEnhanced/mailnotify.tmpl
 1032  wget https://twiki.org/p/pub/Codev/MailNotificationEnhanced/TWiki.pm.diff
 1033  wget https://twiki.org/p/pub/Codev/MailNotificationEnhanced/TWiki.cfg.diff
 1034  cd ..
 1035  cp patches/mailnotify bin/mailnotify
 1036  diff patches/mailnotify.tmpl templates/mailnotify.tmpl
 1037  cp patches/mailnotify.tmpl templates/mailnotify.tmpl
 1038  cp patches/MailNotify.pm lib/
 1039  cat patches/TWiki.cfg.diff
 1040  patch < patches/TWiki.cfg.diff lib/TWiki.cfg
 1042  more patches/TWiki.pm.diff
 1043  patch < patches/TWiki.pm.diff lib/TWiki.pm
 1044  more lib/TWiki.pm.rej

testwiki$ more lib/TWiki.pm.rej
***************
*** 470,475 ****
      # TWiki concept regexes
      $wikiWordRegex = qr/[$upperAlpha]+[$lowerAlpha]+[$upperAlpha]+[$mixedAlpha
Num]*/;
      $webNameRegex = qr/[$upperAlpha]+[$lowerAlphaNum]*/;
      $defaultWebNameRegex = qr/_[${mixedAlphaNum}_]+/;
      $anchorRegex = qr/\#[${mixedAlphaNum}_]+/;
      $abbrevRegex = qr/[$upperAlpha]{3,}/;
--- 472,478 ----
      # TWiki concept regexes
      $wikiWordRegex = qr/[$upperAlpha]+[$lowerAlpha]+[$upperAlpha]+[$mixedAlpha
Num]*/;
      $webNameRegex = qr/[$upperAlpha]+[$lowerAlphaNum]*/;  
+     $webNameRegex = qr/[$upperAlpha]+[_$lowerAlphaNum]*/;
      $defaultWebNameRegex = qr/_[${mixedAlphaNum}_]+/;
      $anchorRegex = qr/\#[${mixedAlphaNum}_]+/;
      $abbrevRegex = qr/[$upperAlpha]{3,}/;

-- MartinCleaver - 19 Nov 2003

Yes Martin, webnames have always been based on a regex in TWiki (this is not my idea but I like it very much ), not exactly the same as Wikinames. I just allow "_" in web name so to be linked, see more in detail the diff ... Note that web names auto-link is mainly used in skins... but is used and still usefull !

I have uploaded a new MailNotify.pm with a fix for a too early closing loop on TWiki users to notify. This is a side effect of filters, job must be done for all users to notify as the topic list is not the same for all.

-- PatrickNomblot - 20 Nov 2003

I encountered some problems after adding new people to the notification list. There is a smal bug in the MailNotify.pm script. After the check if a mail message is empty the routine returns in stead of skipping the send part.

-- PimVanZutphen - 15 Jan 2004

You are right... i did more test let this last version running and look closer to check no more bugs where found. This las version seems to be correct now. Hope you will appreciate

• TWiki.pm pach contains minmum needed for this enhancement.
• did some little refactoring of this Topic.
• added chapter MailNotificationEnhanced

-- PatrickNomblot - 18 Mar 2004

First off, nice work. Now, to the question: How do child topics work? If I have a web "Foo", a topic "Bar", and a child of "Bar" called "Baz". I thought IN{TOPIC="Bar\..*"} would catch changes to Baz since Baz is really Foo.Bar.Baz. To my surprise, Baz is really Foo.Baz but there's some place which records the parent link to Bar. For my usage, I really would like to get notified on a topic an all it's children. If someone could explain how to get at the parent-child data, I'd be happy to do the work (maybe it's a new keyword, IN{PARENT="foo"} )

-- KellyByrd - 27 Mar 2004

IN{TOPIC="Bar"} will only grep on the name of topics. Topics childs name is not join with topic parents in TWiki, it is just a metadata in topic text that you can change while editing topics. This is a new feature that we could add in Topic notification : IN{PARENT="Bar"} Than's for this idea !

-- PatrickNomblot - 08 May 2004

Patrick - I suspect there is significant overlap of this with NotificationPlugin. Do you agree? I am interested in ensuring that the maximum number of people use a single implementation so that the precious resources of the TWikiCommunity are put to efficient use and we don't end up stepping on each other's toes.

-- MartinCleaver - 08 May 2004

Patrick, Martin - I have uploaded my contribution to this notification system mess : YetAnotherNotifySystem It seems clear that the current system deserve a rework. But we need to define the use cases and the feature list. Patrick system, while being a good job, seems to be to much complex for novice users. I tried to simplify it in my contribution.

-- NicolasDuboc - 10 May 2004

Patrick --

I've just begun to use your mail notification on my twiki installation at home -- and I have a problem with the mailnotify.tmpl because it includes two variables %WARNING1% and %WARNING2%. From the MailNotify.PM I can see that these two variables point to topics that don't exist, and I think that is confusing to my users ...

       my $warningText1="You can specify TOPICS to exclude from mail notification for each web in $scriptUrl/view$scriptSuffix/$webNameEnc/WebNotifyExclusion";
       my $warningText2="You can specify TOPICS to exclude from their name, content or author values, see  $scriptUrl/view$scriptSuffix/TWiki/WebNotifyFilter";

I like the idea of these topics, but can you explain to me, how do the WebNotifyFilter and WebNotifyExclusion topics work ?

-- KeithHelfrich - 26 May 2004

Also - Patrick - the email is sent without a logo, and instead of a graphic all we can see is the 'alt' text which reads "[click on the %WIKITOOLNAME% logo to go to homepage]". How is it possible to display an image of the logo instead ?

-- KeithHelfrich - 26 May 2004

• WebNotifyExclusion is jus a list of topics to exclude from notification for the corresponding WEB, for all users.
• WebNotifyFilter is a per user/group filter to exclude or include some topics from notification but also to get notification mail with a specific format.

Please, see above examples and usage for details.

I would need some help to get nicer mails templates as it seems that, by default, mail client display a limited HTML.

-- PatrickNomblot - 30 Jun 2004

Hi Patrick, I have some problems - maybe you can help ? I cannot yet determine whether the DoubleViewInWebNotifyURL is caused by MailNotificationEnhanced, or perhaps by the default code delivered with TWiki. Could you take a look and tell me what you think ? Merci !

-- KeithHelfrich - 15 Jul 2004

Yes, there is a bug in fixing relative URL-s. This bug exist from originate TWiki mailnotify. Please, upload version 06 Aug 2004 which fix this problem.

Thank's

-- PatrickNomblot - 06 Aug 2004

Patrick, a new question for you. After the upgrade to CairoRelease & simultaneous move to SuSE Linux, I'm now having problems with UninitializedValueInMailNotificationEnhanced. Please help, thanks !

-- KeithHelfrich - 22 Dec 2004