Using IMAP Storage for TWiki

Focus on using IMAP as Storage for TWiki.

Related Topics:

• CopyToEmail: Focus on twiki support for getting copy of topic (and attachments) by email.
• SubmitTopicsByEmail: Send email to twiki system so email thread becomes a new topic. Also see: MailInAddOn.

Table of Contents:

What is the key idea here?

Model the store as IMAP folder. i.e. RCS should use IMAP calls to access the topics.

How will each mail look like?

• From: Whoever edited.
• To: Whoever should be the recipients through topic notification. Can be a list of users, group in twiki, or none.
• Subject: [Companywiki.CodingGuidelines/1.0] New coding guidelines uploaded ...
  • Subject line to contain topic name, version and similar metadata.
  • Remaining field will be the "Subject of change" i.e. what got changed on page. We really miss this in shared collaboration due to lack of dedicated form field.
• Body: text/plain, or MIME formatted to include the attachments. (We can make it multipart, and include a HTML version as well so email users can view it easily.)

Why we would like this?

Key Motivation

For users

• Make the data available offline - on laptops. Users can sync the email folder and get latest updates of topics. (They will miss the navigation though.)
  • There is nothing that beats Email today. Client side UI is fast and optimized, and can't be replicated by web interface.
  • There is assumption here that email access controls can be used to give access only to specific webs. Further, if we require topics' access control to be used, we will require some kind of IMAP proxy. This can be an independent product.

From Deployment Angle (i.e. for System Administrators who manage corporate twiki)

• IMAP servers are already available; Email being one of the central services, it is already well managed with existing management infrastructure.
• Existing Administrators can also use email interface to do some content management (such as restoring topics - by deleting latest version, access controls - by adding new members to From/To list and saving the new mail to folder). No need to learn twiki.

From technical angle

• Multiple twiki installations can use email storage simultaneously.
  • Version control should be enhanced to use branches.

• Various connectors can independently manipulate information and do value add. For e.g. database plugin may refresh a table populated from database completely independent of twiki. (Browser users will do a simple 'refresh'.)

• Standardization: Provide web-services API over email storage. This (storage only, supporting versions) API can then become a widely accepted standard, and with semantics to allow independent updates from multiple locations.

Details

Every new save will produce a new version of file. yes, the content will be replicated - including attachments.. If necessary, there can be some macros which will say "Delete after x days", and the interface will delete the older versions after saving the new draft.

Disk storage will be a cache of latest copy. (Just like what RCS does today.)

Standard operations:

• Viewing: The latest copy will always reside on the disk, and that is what would be viewed.
• Search: Will use copy on the disk.
• Diff: Will have to be implemented. (Perhaps it already exists in included RCSwrap.pm.)

And ...

• Lots of storage required. Thankfully, it is not an issue at all.
• Access controls on email will, by default, not be available. But some IMAP servers, such as cyrus, support access control. We will have to implement synch logic between sync twiki and cyrus.
• Imap servers will work well on search for only few headers. We will have to optimize that part properly.
• Being able to update topic from within email: In theory, this should be feasible. But not trivial.

So, this could be quite a feasible option.

-- VinodKulkarni - 02 Dec 2003

Discussions

(Created this topic from discussion in CopyToEmail - VinodKulkarni, 4 Dec 2003)

I was actually thinking of writing a simple server that looked to the client as if it were an IMAP server, but underneath was acutally just talking to the TWikiRcs backend. That way I would have access to the TWiki through my email client, but would not need to replicate the data to another offline format.

In a similar way to exposing the TWiki using WebServices, WebDAV, and with some kind of ODBC / OLEDB / (whatever the .NET thing is) interface.

-- SvenDowideit - 04 Dec 2003

I have scanned some of IMAP servers for this very functionality:

• Cyrus IMAP server: Very difficult to change backend; quite tightly integrated with file formats.
• Apache James project: Very likely candidate (and in Java), but it was kind of suspended project. IMAP protocol is quite clumsy i.e. too many things to take care of.
• Courier IMAP: (Forgot the exact details, but I remember having rejected it.)
• Any in perl? I am not aware.

In essence, you want IMAP server that is modular as well.

But the IMAP client side codes are available in almost all languages: C-Client for C, perl's Mail::Box, and JavaMail. And IMAP servers today are solid and abundant. All ISPs would have an email server installed. Since we don't mind emails replicated in everyone's mailbox, we shouldn't have to mind wiki topics replicated as well.

In fact, offline capability is important requirement today. Wiki as extension of email is a very important turning point, and I think industry will embrace it immediately. (After all, email has remained without much innovations in spite of being no. 1 internet application, right?)

-Vinod