Captured from an email thread. Will summarise later.

---

I've been digging a bit into what would be needed to permit plugins to add attachments to a topic - without requiring update permission on the topic. - without updating metadata - without notifying of topic changes.

It appears that a simple addition to TWiki::Func::saveAttachment has possibilities.

When Func::saveAttachment calls {store}->saveAttachment, the session user is passed for use by store. It appears that if the user is not passed, the store routine bypasses security checks and updates to metadata. In my local testing, DirectedGraphPlugin appears to continue to function, but without the updates to metadata.

I'm most likely missing something - unintended consequences are sure to apply... So what am I missing?

I propose adding a parameter or option to Func::saveAttachment that would cause it to pass a null string in as the user. Possibly adding an option to the data hash anonymous=1? Or alternatively adding a parameter to Func::SaveAttachment to request an anonymous attachment.

After some feedback here I'll open a bug item for a proposed enhancement.

-- GeorgeClark

I presume 'not passing a user' is not already a useful solution to your problem - but I wonder why ..

Also, If you do want to have 'anonymous' Application specific attachments, I think you should consider setting the user to something like 'DirectedGraphPlugin' - so that you're not surprising admins, and can separate your attachments from those of other Plugins.

-- SvenDowideit

Actually, not passing a user would be very useful, but Func::saveAttachment does not offer that option. Hence my proposal for an addition to the Func::saveAttachment.

There is a bit of history here, but the plugin used to write to /pub directly. Since writing to pub is documented as deprecated, I changed it to use the Attachment API. Unfortunately that causes some issues, especially when the graph content is dynamically generated. "viewing" a topic can cause the plugin to update attachments, cluttering revision history, and generating undesired notifications of changes.

If I understood some prior discussions, Peter and Crawford seem to be leaning toward moving back to direct /pub (or other web accessible) I/O, and deferring the TWiki attach API for some future update. I was trying to avoid that, or at least making it an option.

Discovering a very minor change to TWiki::Func::saveAttachment might resolve the issues, it seems too simple. If anonymous attach requested, omit user parameter from call to store.

I think you are right though, rather than totally hiding the attachments by omitting the user parameter which will be confusing to admins. The more complex solution would be to add parameters that would bypass permission checks and set as "minor" to avoid notification, while still adding the attachments to the metadata.

-- GeorgeClark

Sorry, I don't think this is a good idea, for reasons I guess I didn't explain clearly enough before:

1. For any other store implementation (e.g. a database) you cannot add an attachment "without updating metadata". There /happens/ to be a mechanism in the current text file cache that duplicates information on attachments by embedding text into the topic cache, but you cannot assume that will hold for other stores. Most stores will only record a single copy of attachment metadata, and it will be indivisible from the topic's container-nature.
2. You cannot assume that http://.../pub is a directory. Pub URLs are likely to be served by a script in the future (indeed, if you use viewfile they already are) and access-controlled accordingly. If you allow the storage of "invisible" attachments, then you end up with a maintenance nightmare, as Sven suggests. I do not consider creating "fake users" (such as DirectedGraphPlugin) to be a good solution because of the knock-on effects on the store (suddenly attachment change histories are full of users who are not in the user DB - urgh)
3. As I understand it you do not need histories for the generated files you are attaching.

What you describe above sounds to me like a bug. Func::saveAttachment should default to the current user if a user is not specified. I suspect I left it like that because that is how it worked before, and at the time I didn't want to risk breaking anything more, but it sounds to me like the spec needs to be cleaned up to eliminate this misfeature, rather than by hardening the bug into the spec.

I introduced the plugins work area to redirect away from using attachments and "side files" for plugins, and towards a more structured solution. If the work area is published on the web, then you can use it for the sorts of generated files you are working with. Rather than hacking around the attachment API, I would far rather we used a structured solution that supported the concept of web-visible non-twiki files (a sort of "/pub/work/").

-- CrawfordCurrie

ImagePlugin and ImageGalleryPlugin have the same problem. That's why they currently store images under /pub/images/web/topic/...

-- MichaelDaum

Just had a discussion with Sven on #twiki about this. I tend to think of what ImagePlugin and ImageGalleryPlugin (and now potentially DirectedGraphPlugin) do as "creating caches", and as such want to keep that away from the store. But as Sven points out, that cached data is useful in a distributed environment, where several servers may require access to the same caches, and by separating them from the store we make that harder. We are in effect creating a "second tier" store.

It may be better extend the store API to support properly hidden, anonymous and unversioned data, to support distribution of such cache data. But if we do that, it needs to be properly thought through, or we risk making a change that cripples us later.

-- CrawfordCurrie

In a TWiki rendering cluster you'd want to make each node as autonomous as possible, reducing the need to sync across the net as much as possible. Therefore, it is okay for each node to generate thumbnails by itself. They are synced via the content they are generated from anyway.

-- MichaelDaum

There are two use cases for the DirectedGraphPlugin.

1) Topic explicit data. The plugin generates one or more file types. Names can be explicit, or dynamically generated. Access to attachments is useful and the ability to manage them from the attachment list is desirable. For example, a topic describing a protocol flow might generate a ladder flow diagram using the language. The PNG or JPG file might be inline, but other attachment types might be postscript, printer specific (canon, hpgl, ...). It might be useful to look at versions of the old attachments. The only time an attachment will change is if the topic itself is changed.

2) Data derived external to the topic. Attachments might change for every view as the world data changes. The person viewing the topic might not have permission to change the topic. Storing as visible attachments might be useful for management, but probably the more common scenario is inline display of the image without need to manage the attachment. FamilyTreePlugin falls into this second case. As does the DirectedGraphWebmapPlugin I assume.

Obviously I'd prefer that there be a way to accommodate all of this using the store API. If the only use case was inline images, then it's mainly a cache problem. But because it can now generate many different file types for later access, I'm not convinced that it is only a cache issue. Or maybe I'm not understanding the technical definition of cache.

-- GeorgeClark

Good summary, George, thanks. Your remarks, and other inputs, have raised a number of additional questions in my mind. Here's my summary of use-cases for web-visible attachments identified to date. Please forgive me for labouring this, but I need to understand all the angles, and I'm rather simple so it takes a while.

1. Static file explicitly attached to a topic by a user.
   • File must be versioned, unhidden (unless otherwise requested by the user) but otherwise untouched by TWiki.
2. Static file exclusively managed by an extension from private content, independent of topic content
   • e.g. the image and store files used by TWikiDrawPlugin,
   • A change to the attachment does not imply a change to the topic
   • Extension must be able to control if files are hidden. Such attachments will usually be versioned.
3. Static file generated by an extension (eg in a beforeSaveHandler) dependent on topic content
   • e.g. the image files used by ImageGalleryPlugin, DirectedGraphPlugin, format alternatives (.eps, .xcf, .dwg etc)
   • Attachments that change when the containing topic changes is an interesting case. At the moment, attachments are "usually" refreshed by plugins in the commonTagsHandler, because there is no way to hook a handler into the topic change event. As a result, attachments that are generated from included content will not change to reflect changes to that content. This is the same event that is required by cache implementations.
   • Extension must be able to control whether files are hidden, versioned
4. Dynamic file generated by an external process
   • e.g. a cron job on a remote system that is monitoring system status.
   • A change to the attachment does not imply a change to the topic.
   • Files will usually be unversioned. User must be able to choose if file is hidden or not.

So, what's missing from what we already have?

1. The handler I described in (3),
2. The ability to set an attachment as unversioned.

George, your "anonymous change" is really just a hack to achieve (2), but isn't a clean solution. I would suggest:

* add a 'U' attribute that would tell the store that the attachment is unversioned. The topic would be changed once to add the attachment, but later changes would not modify the topic if the attachment is already there. * I don't like the idea of adding "plugin pseudo-users", and would prefer that all attachments are always owned. So the ability to pass "undef" to TWiki::Func::saveAttachment should be removed.

-- CrawfordCurrie

really, the fully scalable solution will be to have a request update queue, similar to the .changes file that can be used by another processing agent to trigger a re-evaluation. Same issue for search index's - not the sort of thing you want to limit to a cronjob, nor to have block a user's current transaction.

Once you contemplate what we have to do to enable separable processing queues to scale TWiki's to 'internet' size I think the architectures become simpler and clearer - and much more asynchronous.

-- SvenDowideit

What about AutoAttachPubFiles, these are added to the meta data w/o user information as well.

-- MichaelDaum

yeah, AutoAttach is a worry - both because of how it works, and also because of the rather proportionality large performance hit it creates.

-- SvenDowideit

I see Auto Attach as a special case of use case (2). In fact, now i think of it, auto-attach ought to be handled in that way - by an extension to the store - rather than as a core function.

-- CrawfordCurrie

due to the performance issues it has, I wrote a plugin that does the job in a controlled fashion -

UpdateAttachmentsPlugin

-- SvenDowideit - 21 Aug 2008

Would be good to see an appropriate hook point in the core, and auto attach/update attachments factored into plugins that use it.

-- MartinCleaver - 21 Aug 2008

> 3.  Ability for a plugin to attach a file without permission checks.
>   

Hmmm. I'm not sure I like the sound of that. Under what circumstances would that ever be necessary?

>>     * add a 'U' attribute that would tell the store that the attachment
>>       is unversioned. The topic would be changed *once* to add the
>>       attachment, but later changes would not modify the topic if the
>>       attachment is already there.
>>     
>
> I understand - treat it like the "h" hidden attribute?
>
> Would this also ask store to not create the ",v" files in the attachment
> directory for an unversioned attachment?  They can get pretty big.
>   

That's up to the store implementation. I would hope that the current RCS store could be made smart enough not to create RCS histories, though.

-- CrawfordCurrie - 21 Aug 2008

Consider FamilyTreePlugin.

• You as an authorized updater add that long lost son to your grandfather.
• I can't update, but I view your great-grandfather, and the plugin has to create a new family tree because of the change.

But since I can't update, the plugin can't attach the updated tree. I can't view the tree from any topic that has not had it's "cache" refreshed with the new family tree by someone with update permission.

An authorized change from outside of the topic has resulted in a change to generated data that will cause a change to the "viewed" topic attachment. The "view" script passing view permissions causes an update through no fault of viewer..

-- GeorgeClark - 21 Aug 2008

What about storing that kind of content in the plugin working directory, modify the plugin to produce a URL that uses viewfile instead of https://twiki.org/p/pub and modifiying viewfile to be able to receive a parameter stating that the file to be seen is not in the pub directory?

-- RafaelAlvarez - 21 Aug 2008

Do other DMS have such features? Is TWiki a DMS?

-- FranzJosefGigler - 21 Aug 2008

TWiki "can work as" a DMS. It lacks the checkout-checkin features of some.

Anyway, now that I think about it, the original requirements are the ability for plugins to update an attachment without:

1. Changes to the Metadata
2. Notification of the topic change
3. Requiring update permission on the topic.

(1) is just not possible: attachment info is metadata (even if Store is modified so the only info stored is which file goes to which topic). (2) May make sense, and need the modification of some other places in TWiki besides the Plugin API. (3) Will possibly open a door for some security problems. What most systems do in this case is to operate under an internal "system" user which has permission for this kind of operations. This also would require an update to the Plugin API.

-- RafaelAlvarez - 21 Aug 2008

For No. 3 the ability to store attachments without validating against the "view" user permission would be a plugin API change/addition. I would hope that it is not exposed as part of the normal user driven file attachment dialogs. And should never be a default. We might also want to make it a bin/configure setting. "Permit plugins to bypass attachment security". Same for the "Un-versioned attachments. As these things effect security, they should probably be in the configure and default to disabled.

I would hope that the only security issues that should be exposed would exist if the user could install or modify the plugin code itself.

-- GeorgeClark - 22 Aug 2008

Rafael, (1) is possible. As an experiment I just added an unversioned attribute to a copy of 4.2.2, and it works - once an attachment has been attached, subsequent updates to that attachment do not modify the topic. (2) is also possible. Suppressing notification has never worked very well (witness the recent flurry of changes in Codev) and needs some brainpower applied to it. (3) I'm not keen on adding more configure options (there are already too many) but the API caller should have some hoops to go through before permitting this.

-- CrawfordCurrie - 22 Aug 2008

Crawford, we agree on that (1) is not possible for new attachements, what you did was to modify Store "so the only info store is which file goes to which topic" As for (3), a badly-coded plugin may allow unwanted code in the attachment area, without any traceability.

I really wish we could use silent notifications...

-- RafaelAlvarez - 22 Aug 2008

Actually a badly-coded plugin can use the deprecated Func::getPubDir and do direct file I/O. The DirectedGraphPlugin (not badly coded - but written before the attachment API was available) used to leave all it's old graphs in /pub, named using a md5sum in the filename, and they were invisible unless using the Auto Attach function. Every modification of a <dot> graph resulted in another new attachment. It could create quite a mess. But at the same time it didn't run into the permission problems, or the excess change notifications.

-- GeorgeClark - 23 Aug 2008

Oops... In looking into the old versions, DGP has always used saveAttachment, marking them hidden. I had a discussion tonight on #twiki about performance of DGP. It appears that rcsLite might be pretty slow with attachments. Not proven, but thats the current assumption. I must have been confused with some other plugin regarding direct file I/O. It would be interesting to know if unversioned attachments might speed up handling. Crawford, did your change also prevent updating of the "png,v" file in the /pub directory along with avoiding the update to the txt,v of the topic itself?

-- GeorgeClark - 24 Aug 2008