TWiki should offer an API for storing and retrieving data so that plugin authors do not need to reinvent the wheel.

Proposed API to store data:

• TWiki::Func::savePluginDataStore( $pluginName, $web, $topic, $id, $data );
• TWiki::Func::readPluginDataStore( $pluginName, $web, $topic, $id, $rev );

savePluginDataStore() saves data. The data to be saved can be uniquely identified by plugin name, web name, topic name and an arbitary id. The plugin must supply the plugin name, but could omit other identifiers, e.g. a plugin could call =TWiki::Func::savePluginDataStore( "SamplePlugin", "", "", "lastupdate", $data ) if the web and topic are not relevant.

TWiki::Func::readPluginDataStore() reads data and returns it back to the plugin. The same identifiers must be used to get content back. Content is under revision control. The head revision is returned if $rev is omitted.

Open questions:

• All content treated as binary?
• What if content needs to be visible by the browser (e.g. an image)?
  • Does the plugin read the data and passs it back to the browser on the fly?
  • Or should all content be available by URL, e.g. /twiki/pub/_datastore/$pluginName/$web.$topic.$id
• Do we need to offer also an API for passing a temporary file name instead of $data as function parameter?
• Do we need revision control? Probably could be omitted in the first version.
• How is the revision number increased? With each save? In sync with the topic revision?

-- PeterThoeny - 06 Apr 2002

As I mentioned in the MathModePlugin topic, sometimes you may need to store files in the preview phase, without affecting the saved version. Probably it would be interesting to have a function for generating a inique dir into the pub dir.

Another point concerns the possibility of having a cache, in order to save time when viewing the pages. I suggest that plugins should notify if they process static contents (like the smiles plugin, the math plugin, etc) or dynamic ones. The same for the standard rendering phases. In this way a cache mechanism could pass the topic to static plugins in the save (or preview) phase, store the partially processed page in the cache directory and then pass it to plugins for dynamic processing in the view phase.

Finally, I would like to have access control also for files in the pub directory (attachments). I'm using TWiki as a front-end for generating quizzes for students, and the database of questions is attached to the page. I would like to avoid the possibility of downloading the database if one does not have read access to the page.

See some proposals about it on AreWebsTopics

-- FrancoBagnoli - 06 Apr 2002

I think a generalized api is a great idea, though we will face a decision on where we want to be on the simplicity vs capability spectrum.

Lately I have been looking for a mechanism that allows me to store data on topic save to some type of database (probably something very simple like SQLite which is file based). However all of the data would go into one database file per web regardless of topic. So for my own uses, an api that gives me a filehandle or returns the path to file that I can modify would be best for me. That way I could use one db file per web.

-- JohnCavanaugh - 06 Apr 2002

A plugin data store sounds like a great idea. Considering KISS and existing TWiki capabilities I would suggest:

• Store data in file attachments
  • A topic would use _Plugin_<PluginName>
  • A web would store store under WebPreferences (or similar)
  • Whole of TWiki would be stored under the Plugin topic
• This would mean it was autmatically under version contol that was linked to the topic
• This attachment could be marked as hidden
• Normally a plugin would just use one attachment, but if it need separately addressable URLs then more than one could be used

The above approach would require very little coding. I'd rather abstract the file system away from the Plugins.

-- JohnTalintyre - 09 Apr 2002

Data store for plugins is a great idea.

Seeing how the plugin name can't be optional, how about making it an object reference on which the operation is invoked?

$plugin = $new TWiki::Plugin; $plugin->saveDataStore( $web, $topic, $id, $data );

-- MartinCleaver - 10 Apr 2002

The approach of using attachments that I mentioned above would have potential locking problems for web and TWiki wide settings. This could be avoided initially be either:

• not providing these facilities
• updating the attachments but not the topic they live with
• having special topics with locks that are only kept open briefly - this would likely mean limited update access e.g. limited to admin group
• be able to update the attachment even if the topic is locked - a simple option but potentially confusing

-- JohnTalintyre - 10 Apr 2002

Very good idea to use the existing file attachment feature for the plugin store.

Locking should not be a problem. Already now it is possible to upload an attachment to a locked topic because the topic save algorithm reads the meta data from disk, merges it with form data if present, then saves the topic. For the data store, file attach should be enhanced to save the lock status away (locked? by who? when?), do the attachment update, then restore the lock status.

Where to store? The most simple way is to attach the files to the plugin topic itself. For example, TWiki::Func::savePluginDataStore( $pluginName, $web, $topic, $id, $data ); would store $data to the $pluginName topic with name _PD_$web_$topic_$id. One issue is how to handle access control, i.e. if the user running the plugin has no permission to update topics in the TWiki (or Plugins) web.

Or we could generalize this to allow plugins to maniplute files attached to any topic:

• TWiki::Func::saveAttachment( $web, $topic, $data );
• TWiki::Func::readAttachment( $web, $topic, $rev );
• TWiki::Func::getAttachmentList( $web, $topic );

This could be a potential security hole. TWikiAccessControl needs to be considered.

-- PeterThoeny - 11 Apr 2002

Would there be anyway to turn off versioning for the api?? I can definitely imagine scenarios where it would be just a computation/disk waste to track versioning of these PluginDataStore attachments.

-- JohnCavanaugh - 11 Apr 2002

Is this already covered by TWiki::Func::getWorkArea ?

-- RafaelAlvarez - 12 Aug 2008

Yup - that, and the attachment API. It's really a duplicate, but there is no such state on the forms so marking as MergedToCore.

-- CrawfordCurrie - 12 Aug 2008