Feature Proposal: S/Mime support for notification e-mails

Motivation

Security. X.509 support

*Wiki notifications are rich HTML - filled with links. In today's SPAM-filled, hostile world, they're subject to being mis-identified.

Further, email in this format could be used for various phishing attacks.

Description and Documentation

Need an option to cause e-mail sent by TWiki to be signed with an administrator's S/MIME certificate.

This makes it easy to ensure the integrity of WebNotify messages, and to SPAM-classify them appropriately.

It turns out that the code to send s/mime signed email isn't as scary as one might think.

In fact, here's a complete patch. The only restriction is that I didn't take the time to figure out how to persuade Net::SMTP to accept the additional headers - sendmail is good enough for me.

Examples

I'd rather not discuss the exploits possible with unsigned e-mail; however, closing those holes is clearly a requirement for software that calls itself 'enterprise class'.

Impact

This should be transparent to code except configure.

For the administrator, it means obtaining a certificate/key pair. The most difficult thing is to get permissions right; the private key MUST be kept secure; the certificate can (and should) be public. This is a bit tricky under selinux, but that's no different from all the other hassles of runnng TWiki under selinux.

Implementation

Here's the promised patch (against 4.2.3 )

cd  www/twiki/lib

lib$ for f in TWiki.spec TWiki/Configure/Checkers/SmimeCertificateFile.pm TWiki/Configure/Checkers/MailProgram.pm TWiki/Configure/Checkers/SmimeKeyFile.pm TWiki/Net.pm ; do if [ -r $f~ ]; then diff -U6 $f~ $f ; else diff -U6 /dev/null $f ; fi ; done
--- TWiki.spec~ 2008-09-11 23:41:58.000000000 -0400
+++ TWiki.spec  2008-12-15 10:26:07.000000000 -0500
@@ -996,12 +996,30 @@

 # **STRING 30**
 # TWiki administrator's name address, for use in mails (first name and
 # last name, e.g. =Fred Smith=) (used in %WIKIWEBMASTERNAME%)
 $TWiki::cfg{WebMasterName} = 'TWiki Administrator';

+# **PATH**
+# Secure email certificate.  If you want e-mail sent by TWiki to be signed,
+# specify the filename of the administrator's X.509 certificate here.  It
+# must be in PEM format.  You must also use a mail program (not Net::SMTP)
+# in the following settings. <em>If you do not want signed e-mail,
+# leave this field blank. </em>
+$TWiki::cfg{SmimeCertificateFile} = '$TWiki::cfg{DataDir}/cert.pem';
+
+# **PATH**
+# Secure email certificate.  If you want e-mail sent by TWiki to be signed,
+# specify the filename of the administrator's X.509 private key here.  It
+# must be in PEM format.  <em>Be sure that this file is only readable by the
+# TWiki software; it must NOT be readable by users!</em>
+# You must also use a mail program (not Net::SMTP)
+# in the following settings. <em>If you do not want signed e-mail,
+# leave this field blank. </em>
+$TWiki::cfg{SmimeKeyFile} = '$TWiki::cfg{DataDir}/key.pem';
+
 # **COMMAND**
 # Mail program. If Net::SMTP is installed, it will be used in preference.
 # To force TWiki to use the {MailProgram}, unset both {SMTP}{MAILHOST}
 # below and all SMTPMAILHOST settings in your TWiki's Preferences topics.
 # This needs to be a command-line program that accepts
 # MIME format mail messages on standard input, and mails them.
@@ -1046,13 +1064,13 @@
 # <b>If you change this setting you will have to
 # use TWiki to manually rename the topic in all existing webs</b>
 $TWiki::cfg{NotifyTopicName}     = 'WebNotify';

 # **BOOLEAN EXPERT**
 # Set this option on to enable debug
-# mode in SMTP. Output will go to the webserver error log.
+# mode in SMTP. Output will go to the webserver error log
 $TWiki::cfg{SMTP}{Debug} = 0;

 # **STRING 30 EXPERT**
 # Some environments require outbound HTTP traffic to go through a proxy
 # server. (e.g. proxy.your.company).
 # <b>CAUTION</b> This setting can be overridden by a PROXYHOST setting
--- /dev/null   2008-04-26 06:27:25.709194980 -0400
+++ TWiki/Configure/Checkers/SmimeCertificateFile.pm    2008-12-15 10:15:12.000000000 -0500
@@ -0,0 +1,37 @@
+#
+# TWiki Enterprise Collaboration Platform, http://TWiki.org/
+#
+# Copyright (C) 2000-2006 TWiki Contributors.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version. For
+# more details read LICENSE in the root of this distribution.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# As per the GPL, removal of this notice is prohibited.
+package TWiki::Configure::Checkers::SmimeCertificateFile;
+
+use strict;
+
+use base 'TWiki::Configure::Checker';
+
+use TWiki::Configure::Checker;
+use TWiki::Configure::Load;
+
+sub check {
+    my $this = shift;
+
+    my $certFile = $TWiki::cfg{SmimeCertificateFile} || "";
+    $certFile =~ s/%DATE%/DATE/;
+    TWiki::Configure::Load::expandValue($certFile);
+    my $e = !-r ( $certFile ) && "Can\'t read $certFile";
+    $e = $this->ERROR($e) if $e;
+    return $e;
+}
+
+1;
--- TWiki/Configure/Checkers/MailProgram.pm~    2008-09-11 23:41:58.000000000 -0400
+++ TWiki/Configure/Checkers/MailProgram.pm     2008-12-15 11:01:38.000000000 -0500
@@ -22,27 +22,37 @@

 use base 'TWiki::Configure::Checker';

 sub check {
     my $this = shift;

-    return '' if( !$Twiki::cfg{EnableEmail} );
+    return '' if( !$TWiki::cfg{EnableEmail} );

     eval "use Net::SMTP";
-    my $n;
+    my $n = '';
     my $useprog = 0;
+
+    my $signmail = 0;
+    if( $TWiki::cfg{SmimeCertificateFile} || $TWiki::cfg{SmimeKeyFile} ) {
+       $signmail = 1;
+       unless( $TWiki::cfg{SmimeCertificateFile} && $TWiki::cfg{SmimeKeyFile} ) {
+           $n = $this->WARN( "Signed e-mail requires both a certificate and a key file.");
+       }
+    }
     if ($@) {
-        $n = "Net::SMTP is <b>not</b> installed in this environment. ";
+        $n .= "Net::SMTP is <b>not</b> installed in this environment. ";
         $useprog = 1;
     } elsif( !$TWiki::cfg{SMTP}{MAILHOST} ) {
-        $n = $this->WARN('Net::SMTP is installed in this environment, but {SMTP}{MAILHOST} is not defined, so the {MailProgram} <b>will</b> be used..');
+        $n .= $this->WARN('Net::SMTP is installed in this environment, but {SMTP}{MAILHOST} is not defined, so the {MailProgram} <b>will</b> be used..') unless( $signmail );
         $useprog = 1;
     } else {
-        $n = $this->NOTE('<em>Net::SMTP is installed in this environment, so this setting will <b>not</b> be used.</em>');
+        $n .= $this->NOTE('<em>Net::SMTP is installed in this environment, so this setting will <b>not</b> be used.</em>');
+       $n .= $this->WARN('<em>Signed e-mail is not supported by Net::SMTP.  If you want signed e-mail, please leave {SMTP}{MAILHOST} blank.  If not, please leave {SmimeCertificateFile} and {SmimeKeyFile} blank.') if( $signmail );
         $useprog = 0;
     }
+
     if ($useprog) {
         my $val = $TWiki::cfg{MailProgram} || '';
         $val =~ s/\s.*$//g;
         if( ! ( -x $val ) ) {
             $n .= $this->WARN("<tt>$val</tt> was not found. Check the path.");
         }
--- /dev/null   2008-04-26 06:27:25.709194980 -0400
+++ TWiki/Configure/Checkers/SmimeKeyFile.pm    2008-12-15 10:14:56.000000000 -0500
@@ -0,0 +1,37 @@
+#
+# TWiki Enterprise Collaboration Platform, http://TWiki.org/
+#
+# Copyright (C) 2000-2006 TWiki Contributors.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version. For
+# more details read LICENSE in the root of this distribution.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# As per the GPL, removal of this notice is prohibited.
+package TWiki::Configure::Checkers::SmimeKeyFile;
+
+use strict;
+
+use base 'TWiki::Configure::Checker';
+
+use TWiki::Configure::Checker;
+use TWiki::Configure::Load;
+
+sub check {
+    my $this = shift;
+
+    my $certFile = $TWiki::cfg{SmimeKeyFile} || "";
+    $certFile =~ s/%DATE%/DATE/;
+    TWiki::Configure::Load::expandValue($certFile);
+    my $e = !-r ( $certFile ) && "Can\'t read $certFile";
+    $e = $this->ERROR($e) if $e;
+    return $e;
+}
+
+1;
--- TWiki/Net.pm~       2008-10-24 07:09:22.000000000 -0400
+++ TWiki/Net.pm        2008-12-15 11:07:27.000000000 -0500
@@ -341,20 +341,45 @@
     # split up header lines that are too long
     $addrs =~ s/(.{60}[^,]*,\s*)/$1\n        /go;
     $addrs =~ s/\n\s*$//gos;
     return $addrs;
 }

+sub _slurpFile( $ ) {
+    my $file = shift;
+
+    unless( open( IN, '<', $file ) ) {
+       ( $<,$>) = ( $>,$<);
+       die( "Failed to open $file: $!\n" );
+    }
+    my $text = do { local( $/ ); <IN> };
+
+    unless( close IN ) {
+       ( $<,$>) = ( $>,$<);
+       die( "Failed to close $file: $!\n" );
+    }
+
+    return $text;
+}
+
 sub _sendEmailBySendmail {
     my( $this, $text ) = @_;

     # send with sendmail
     my ( $header, $body ) = split( "\n\n", $text, 2 );
     $header =~ s/([\n\r])(From|To|CC|BCC)(\:\s*)([^\n\r]*)/$1.$2.$3._fixLineLength($4)/geois;
     $text = "$header\n\n$body";   # rebuild message

+    if( $TWiki::cfg{SmimeCertificateFile} && $TWiki::cfg{SmimeKeyFile} ) {
+       use Crypt::SMIME;
+
+       my $smime = Crypt::SMIME->new();
+
+       $smime->setPrivateKey( _slurpFile( $TWiki::cfg{SmimeKeyFile} ), _slurpFile( $TWiki::cfg{SmimeCertificateFile} ) );
+       $text = $smime->sign( $text );
+    }
     open( MAIL, '|'.$TWiki::cfg{MailProgram} ) ||
       die "ERROR: Can't send mail using TWiki::cfg{MailProgram}";
     print MAIL $text;
     close( MAIL );
     die "ERROR: Exit code $? from TWiki::cfg{MailProgram}" if $?;
 }

 

-- Contributors: TimotheLitt - 15 Dec 2008

Discussion

Thank you Timothe for the feature request with solution. I added the date of commitment in the form below to start the two week review period, as documented in our TWikiReleaseManagementProcess.

Would you like to commit this change? ReadmeFirst and SoYouWantToBeATWikiDeveloper gets you started.

-- PeterThoeny - 15 Dec 2008

I'd rather have someone familiar with the process handle the commit, which is why I didn't sign up for that part of the job.

I don't have time to learn the ins and outs of your high-overhead process - particularly since at this point, I'd also have to do the same for the fork.

If there isn't anyone else willing to apply the patch files and then do whatever else is involved, I'll keep this as a local patch. That seems like considerably less work - unfortunately for the community.

I'm willing to give improvements back to the project, but I don't have the bandwidth to be a full-time developer.

It's unfortunate that those of us who wish to use (and occasionally contribute) to the project -- must now do twice the work. At least until it's clear which branch to stay with (or move to.)

-- TimotheLitt - 16 Dec 2008

I fully understand, and fully agree with your statements. It was not my choice to fork; they prepared it silently for several month. As expected (and unlike the forkers want you to believe) we have momentum on twiki.org. On TWiki.org we have over 1000 people who agree to the new TWikiCodeOfConduct since we introduced it 1.5 month ago, and 30 who don't. We also had a new release, whereas the forkers have yet to release something.

Someone from the TWiki community will take care of your feature request.

-- PeterThoeny - 16 Dec 2008

Thimothe, we plan to take this into 4.3 release. Please provide some docs in the core code. Best place probably where the SMTP mail setup is documented. You could change doc directly in the TWiki web on twiki.org; we can take care of checking it in.

-- PeterThoeny - 19 Jan 2009

This is pretty well documented in the configure script.

For completeness, I updated the installation instructions to reflect this feature. I did not document all the ins and outs of S/MIME e-mail and certificate management. There are books on the subject, and lots of internet resources. I did document the TWiki feature.

I didn't want to confuse anyone installing 4.2.x but wasn't sure how you handle this. So I forked the topic to TWiki.TWikiInstallationGuide430.

-- TimotheLitt - 20 Jan 2009

This is merged into TWikiInstallationGuide as of TWiki-4.3.0.

-- PeterThoeny - 2009-04-12

I just pulled trunk into a local repository, and this code is missing. Any idea what happened? The last note here says this made it into 4.3.0... and the state says "merged to core"

Since I have a current SVN repository (but am not running it), I merged the patch back in to my local copy.

Attached is a patch file that should apply cleanly to trunk - at least today. There should be no functional change from the original patch.

• smime.patch.trunk.21496.tar.gz: Re-patched against Trunk rev 21496

Thanks.

By the way, a site annoyance: I entered this comment and hit add. I wasn't logged in. I got a login prompt for the "open source" login. My username and password were not accepted.

I logged in via the "account" pulldown at the top right. Now I'm logged in, but I had to retype the text.

This shouldn't happen.

-- TimotheLitt - 2011-06-14

It looks like I did a snafu in 2009. Apparently I only checked in the doc update in the TWikiInstallationGuide, TWikirev:17854, and missed the code update. Thanks for posting the updated S/MIME patch. I'll check this in, tracked at TWikibug:Item6749. Feature proposal re-opened.

Thanks for the heads up on the login issue, i'll check.

-- PeterThoeny - 2011-06-14

I checked in the patch, thank you again Timothe for the patch. I did two changes:

• twiki/lib/TWiki/Contrib/core/MANIFEST had to be updated with the two new files.
• twiki/lib/TWiki/Net.pm had a use Crypt::SMIME; which will be compiled when the module is compiled, resulting in a runtime error in case Crypt::SMIME is not installed. It also slows down the system if it is installed. I changed that to require Crypt::SMIME; so that it gets loaded only when needed.

Timothe, could you please do an svn up and test if things work as expected?

-- PeterThoeny - 2011-06-15

also, the patch contained unrelated {UseModPerl} settings. I omitted those.

-- PeterThoeny - 2011-06-15

Did the svn up, took our changes. There was one glitch - an extra '+' in byte zero of lib/Twiki/Configure/Checkers/SmimeCertificateFile.pm. With it, configure doesn't load.

Delete it and all's well. Sent evidence to you via email.

modperl was not in the original patch or intended, must have been when I ported it to trunk - probably would have reversed something done on trunk in the meantime. Good catch - my test system isn't running mod_perl.

Also, when I hit Add comment, I again got the "you do not have permission to access bin/save/view/Codev/SimimNotificationSupport", despite entering my correct User/Password. This is a plaintext webpage response, not a browser authentication pop-up. And in this case, I used IE8. Hope this helps you diagnose & fix.

Again, logging in by Account->login did work.

-- TimotheLitt - 2011-06-16

(the double post is because add comment hung when I submitted, so there's something else going on.)

-- TimotheLitt - 2011-06-16

Thanks Timothe, good catch. Is now fixed in SVN trunk.

-- PeterThoeny - 2011-06-16

I did some additional fixes, see TWikibug:Item6749 - main item was to make the SMIME settings optional - configure was complaining if they were empty.

-- PeterThoeny - 2011-06-20