Plugins API Policy discussions

Discussions refactored out of the PluginsApiPolicies

Discussion on common code base
Brief discussion on other API functionality
Back to code base issues
Discussion on deprecating certain usage of readFile and writeFile
Discussion of Kenneth's proposed business rule

Discussion on common code base

I disagree strongly about using a common code base for two versions of the core being considered a "preferred" model. Currently most of the plugins violate the above policies in one or more ways, which makes for fragile and difficult-to-maintain code except in the rare instances where the version-dependent code is self-contained and loaded only when appropriate. The many instances of this can easily be seen by looking at PluginsConformanceReport, which lists violations in great detail.

-- MeredithLesly - 08 May 2006

I have to agree. I have tried to maintain compatibility that way, and find it rapidly turns into a nightmare in all but fairly simple plugins.

-- CrawfordCurrie - 08 May 2006

Perhaps we need a model plugin to show plugins developers to bring their plugins up to snuff (not to mention us'ns)?

If we're going to suggest that people use the same code base, I assume you mean something of the sort:

FooPlugin.pm
FooPlugin
- FooCairoPlugin.pm
- FooDakarPlugin.pm

where FooPlugin.pm does little more than determine which of the modules to load?

-- MeredithLesly - 08 May 2006

I had in mind something like this (in the installer):

unless( -f lib/LocalSite.cfg ) {
   # rename FooCairoPlugin.pm to FooPlugin.pm
   # overwriting the Dakar version

-- CrawfordCurrie - 08 May 2006

On conditional code, this is a usability question. I prefer to make TWiki as easy as possible for the administrators and users. That means sometimes more work for the developer, especially if the APIs get retired and/or enhanced. I am using conditional code in some Plugins I maintain. The conditionals are usually just a few lines of code encapsulated in a subroutine, for example calling a proper Func function if on Dakar or an illegal Store function if on Cairo.

What is more important, customer focus or a shiny Plugins conformance report? I prefer the former and do not mind if the conformance report lists some of my Plugins as not conforming even they are (e.g. not groking the conditional code.)

Using an installer to selectively install one or the other module is an option. It assumes though that admins must use the installer. What about the steps before that? The packager also needs to copy/rename modules selectively.

-- PeterThoeny - 08 May 2006

There is another approach. Put an indirection layer between you and twiki incompatible call. ie:

MyBackwardCompatiblePlugin.pm
MyBackwardCompatiblePlugin
- MyAPI.pm
  - MyAPI
    - Cairo.pm
    - Dakar.pm

And code everything in you plugin against MyAPI.pm. This has two advantages:

All the conditional logic is centralized in one module (MyAPI.pm)
MyAPI can help to identify API extensions.

If there could be a contrib that helps identify the TWiki version (there are many, many ways to do it), that would be great.

-- RafaelAlvarez - 08 May 2006

TWikiPlugins documents the proper way to detect a TWiki version.

-- PeterThoeny - 08 May 2006

If you want people to maintain their plugins, it's necessary to make things as easy as possible for them to do so. And ensuring reliable, well-maintained plugins is focusing on the users of TWiki. In this, as in many other cases, "developer-centric" is "user-centric" as well. I don't know why you think these are inherently in opposition.

You are losing sight of an essential part of the TWikiMission when you sarcastically refer to a "shiny Plugins conformance report". The conformance report (although a simplified one would be much preferable) is an important tool for users who want to ensure a reliable installation. In fact, it's "developer-centric" at its worst when one opts for the easy solution rather than the right one.

-- MeredithLesly - 08 May 2006

Related to some changes I had to do in our storage setup this week, I ran into a snapshot of our install from when we were upgrading from Beijing to Cairo. Plugin zips carefully noted up with which worked where, which weren't yet tried out etc.

So for a while my memory is fresh on the troubles even "little things" summed up to at that point.

So I was thinking - whatever logic is going to be involved in the final solution (for branching per code base); if packaging could be a simple "one zip per plugin" I think we would help many.

Using the installer to customize the plugin at installation time is an insteresting idea (indeed a solution for plugins with skin issues - draw, comment, history etc).

-- SteffenPoulsen - 08 May 2006

"one plugin for each release". Please remember that most extensions developers are not developing for the benefit of twiki.org. They are developing for the benefit of their employers; and they happen to be generous enough to publish the plugins that they write. If an employer has upgraded, then the plugin author no longer has any motive to maintain the Cairo version of the plugin. If you want to create an environment where plugins can be maintained on older versions, then you have to do two things:
1. Create an environment where it is easy to do so. I have tried to do this in the past by providing compatibility modules to ensure new APIs are back-ported to old core APIs, but I didn't get any help or support, so I gave up. It's the wrong approach, anyway. The right approach is to publish rich APIs that reflect the functionality that all plugins authors need access to, so that the need to subvert the API is minimised.
2. Find a way to motivate plugins authors to maintain compatibility. One way might be to publish a "compatibility index" that shows which plugins are compliant with which TWiki versions, with a mechanism by which users can request support for older versions.

***********

In the old days the anyone could make a one line fix and upload a zip to Plugins web and update the TWiki topic to the immediate benefit of all users.

And Peter, the requirement to make plugins dual compatible instead of accepting a Cairo and a Dakar version is also an increased requirement for the plugin developer who may only know 5% of the perl you, Crawford and the other very experienced professional programmers know.

As a user I do not care if a plugin works in all versions as long as I can download a version that works with my installed TWiki version and I have no doubt which one is the right one.

-- KennethLavrsen - 19 Apr 2006

Brief discussion on other API functionality

Refactored from BrainDump, which contained refactored stuff from somewhere else. Thus the date.

By reading ReadmeFirst, I found that it's now a recommendation to run all the commands through the TWiki::Sandbox module. This means that we're adding it to the TWiki API.

-- RafaelAlvarez - 13 May 2006

I hope someone who understand what that means documents it here. It's certainly not currently in ReadMe.

-- MeredithLesly - 13 May 2006

It means that whatever methods are defined in TWiki::Sandbox must be keep in place with the same name and semantic from now on. The same is true for TWiki::Meta, BTW.

-- RafaelAlvarez - 13 May 2006

Meta is mentioned above. I've never used TWiki::Sandbox in a plugin, so I don't have a clue what its relevance is.

-- MeredithLesly - 13 May 2006

Back to code base issues

There are users who have upgraded to Dakar and find that some of their plugins no longer work. They are often willing (and find it necessary!) to put in the time to upgrade to plugin to Dakar, especially since there are directions on how to do that. They have no motive and often not enough expertise to make a plugin work for both Cairo and Dakar, however.

Supporting CurrentVersion/!PreviousVersion is, in general, only a Dakar/Cairo issue, since Dakar introduced a much more functional API. As long as people use the Dakar API (and, obviously, only need what's in the API to accomplish their task), their plugins should continue to work in Edinburgh.

In the end, making things as easy as possible for extension developers even if it involves an extra minute or two of thought on the part of adminstrators will make it more likely that people will contribute to the project. Having more contributions is a win for everyone. We shouldn't put obstacles in the way.

-- MeredithLesly - 16 May 2006

Good point.

-- RafaelAlvarez - 17 May 2006

Discussion on deprecating certain usage of `readFile` and `writeFile`

There's a conflict between the documentation for TWiki::Func and what is provided. Specifically, plugins are supposed to read and write files in their workarea and nowhere else. Unfortunately, there are two problems. First, some plugin authors seem unaware of this change in policy. Second and more importantly, plugin authors are told they are responsible for managing their workarea, but there's no deleteFile.

readFile and writeFile should have been deprecated for 4.0, and readWorkFile, writeWorkFile, and deleteWorkFile should have been added, as they have, in fact, been added to develop. The question is what to do about it now.

It's trivial for any plugin that uses readFile and writeFile in combination with getWorkArea to have that code change to use readWorkFile and writeWorkFile instead. I don't know whether any plugins are actually "managing their work areas" or whether files are accumulating there, given the lack of deleteFile; and, perhaps, the work area isn't being used by most of them in lieu of (incorrectly) writing files to the pub hierarchy.

-- MeredithLesly - 30 May 2006

Why should readFile and writeFile be deprecated? Why wouldn't a plugin be allowed to read a file elsewhere? It could be the entire purpose of this special plugin to read files elsewhere? Maybe integrating with some CNS system or similar.

We are telling plugin authors to use the API to avoid having to rewrite the plugins all the time. And for the admins that install plugins it is horrible having a ticking bomb over your head knowing that next time you upgrade your TWiki, a plugin that many topics depend on will stop working, unless you are lucky that some programmer updates the plugin. And all because there was some developers that changed the API - again! Extending API is not the issue. It is continuous deprecation that is the issue.

If I was a plugin developer and read this topic I would consider not using the API for reading and writing files but write directly using standard perl because obviously you cannot trust that anything will work two TWiki releases from now.

The list of plugins that worked in Cairo and does not work in TWiki4 is still long. And to the admins and the end users this fact is a real problem. Yes, many of those plugins used non-API calls because there simply was no API that did what they needed. At least those plugins that do use API should not stop working. And all the warnings to the plugin authors are no good because 50% of the plugins are abandoned and major bulk of the people that use them have nothing to do with the original author and do not have the skills to fix them themselves. They just downloaded them from Plugins and trusted that TWiki was a stable and professionel well managed open source product that they would also work in the future. And then they upgrade and they are screwed.

Deprecation should happen only when...

There is an important performance issue if you don't.
You are prepared to update ALL the plugins that use the old function to the new API. Or you have agreements from active plugin authors that they will update their plugin.

Remember that at least 50% of TWiki's value and the major part of actual lines of code lies in the plugins.

-- KennethLavrsen - 30 May 2006

The deprecation policy is for major releases, so it is too late to deprecate readFile and saveFile in TWiki 4.

I do not agree that it is OK to break Plugins just for the sake of a nice API. Although it might be useful to add readWorkFile and saveWorkFile, it does not prevent authors from using Perl's open( FILE, "/etc/passwd" ).

Plus, the Plugin author might need to interact with other files on the system, so it is useful to retain readFile and saveFile.

-- PeterThoeny

The documentation for saveFile explicitly states that the function is to be used for the Plugin workarea. The same goes for readFile. While the documentation does not deprecate the functions themselves, it does deprecate using the functions to read and write files in anywhere other than the provided work area. If plugins are doing otherwise, then they are violating API and are therefore not guaranteed to work in future versions.

Many plugins for Cairo have to break commonly understood rules for API because the Cairo API doesn't give adequate functionality. That's why so many plugins that support both Cairo and Dakar have to use conditional code: reaching into the core for Cairo and using API for Dakar. Plugins that do not violate the Dakar API --which also matters such as not assuming that files on disk will remain in the same formats or the same places -- will continue to work without change. This is the point of this topic: to establish the ground rules which, if followed, will ensure that properly written plugins will work in Dakar and in future versions.

I am not setting policy here: I'm merely referencing the official documentation. If you have issues with the official API, take it up with CDot, not me.

-- MeredithLesly - 30 May 2006

One thing I am not on par with yet - if we're deprecating these, what API functions are to be recommended to read/write/delete files outside the TWiki directory?

-- SteffenPoulsen - 31 May 2006

Currently there aren't any. (There never was one to delete a file, btw, which is a glaring omission given that a plugin is responsible for managing its work area). You really should ask CDot about this. (And, hopefully, he'll correct anything I suggest in the rest of this comment that is wrong). But, as a general rule, I'd suggest that you wrap it in a Contrib (or add it to an existing appropriate one) which you then use in your plugin. All contribs are doing, really, is creating API which is then used (in the perl sense as well as the literal sense) by other extensions. As an extra benefit, other people can then use functions from the same contrib, rather than having to reinvent it themselves, assuming that they're even able to.

For example, when I first started working with #twiki, I was working on a plugin that needed some of the functionality that now is in FuncUsersContrib but at that point only resided in the core. CDot was kind enough to pull the bits of code out of the core that I needed and put them in functions in my plugin for me to use. I later took them out of the plugin and added them in FuncUsersContrib, since that was both cleaner and allowed others to use the functions as well. (By "cleaner" I mean that the plugin then =use=d the module and then called functions in it, allowing code in the contrib to be rewritten if necessary without breaking my plugin.) Had CDot not helped me, then I would either have given up on the plugin or done it in a way that may well have broken, either due to unforeseen side-effects or changed implementation.

As an extra benefit to the community as a whole, contribs are far easier than plugins to examine for functionality that should be added to the core.

-- MeredithLesly - 31 May 2006

I tried to deprecate readFile and writeFile in TWiki-4, but was asked not to do so. My argument was that these methods encourage the view of the TWiki DB as files, and are inherently dangerous to have in the API, and they should be deprecates and replaced with the official perl equivalent. I don't like masking file access in functions like readFile and writeFile because it makes the caller so dependant on the implementations. For example, do those functions handle binary files? On all platforms? Are they thread safe? Do they change the values of any of the plethora of $variables that perl uses to control IO? Can you use them on pipes? Will all those assumptions always be true? IMHO it is better not to hide that sort of thing, but to encourage plugins authors to use the real, in-your-face, official perl way of doing things.

Because I was blocked from deprecating them, I rewrote the help for readFile and writeFile to try and encourage their use only on the work area for individual plugins, and discourage their use on files to existing data or pub areas. Clearly I didn't want to compound the problem by adding a delete function, as unlink(TWiki::Func::getWorkArea('MyPlugin').'MyFile') works just as well, and is documented in perldoc -f unlink.

There is nothing to stop plugins authors using standard perl functions such as open and opendir etc. For data in the work area, and for files outside the TWiki content repository, I would positively encourage them to do so. The key point is that these functions must not be used to access TWiki content, as that then ties the calling plugin down to a single implementation of the TWiki data store. As stated in the policies above You cannot rely on files on disk remaining in the same formats or the same places. You cannot rely on the current directory structure. Use the APIs to manage the database.

-- CrawfordCurrie - 01 Jun 2006

Kenneth wrote: "Deprecation should happen only when...

There is an important performance issue if you don't.
You are prepared to update ALL the plugins that use the old function to the new API. Or you have agreements from active plugin authors that they will update their plugin."

Or unless we want to give authors or other maintainers ample warning so that they have enough time to fix the plugins before functionality changes and the plugins break. That's the point of deprecating functionality rather than abruptly removing it. Had Crawford been allowed to deprecate the functions for Dakar, authors would have had considerably more warning regarding functionality that may be necessary in order to improve performance.

"Remember that at least 50% of TWiki's value and the major part of actual lines of code lies in the plugins."

Yes, and that's why solid API and plugin policies, as well as the best tools to measure extension reliability are so important. Mr. Corporate Decision Maker is going to feel a lot more comfortable about using a plugin if there's a good tool to measure its reliability. And the fact is that, as Crawford notes, plugins that don't use the APIs to manage the database are likely to break down the road. Or important improvements may wind up being blocked because too many plugins will break.

-- MeredithLesly - 02 Jun 2006

Thanks, Meredith, for moving this away from the policy document. It was getting rather crowded in there.

-- CrawfordCurrie - 05 Jun 2006

Meredith wrote: "Or unless we want to give authors or other maintainers ample warning so that they have enough time to fix the plugins before functionality changes and the plugins break."

What does that help? Half the plugins were written by prople that no longer maintain them. Probably because they moved on to a different product than TWiki. But that does not mean the plugin is no longer used. The users of a plugin are not the same people that wrote it. If you deprecate a function in the API without fixing the plugins that use it you prevent many TWiki installations using this plugin from ever upgrading. And then you can give oh so long warnings. If a developer absolutely have to deprecate a function then we need to access which plugins use the function and fix those plugins. And if you do not want to spend time fixing plugins - don't break them. It is that simple. Think about the users using the plugin and not about the original plugin authors.

Naturally there may be plugins that are not on TWiki.org and we cannot fix those. But in this case I care less, because if someone takes TWiki and all the work gone into I expect them to give back their plugins as a contribution. And if they chose not to - then - too bad.

It always comes back to the very simple rule. First - think about the users before you change anything. And then after that you can think about the admins. You do not have to think about the programmers too much because they are perfectly capable of helping themselves.

About readFile and writeFile. I think Crawford did a good job with the documentation. It is very clear what to use those functions for and what to not use them for. Even for a perl beginner like me.

-- KennethLavrsen - 05 Jun 2006

Functionality will change. It has to, or TWiki will continue to run like a snail. If a plugin reads from or writes to a file that is no longer a file for performance or other important architectural reasons, it will break, just like plugins that reached into the Cairo core generally broke.

If TWiki is too slow to use, it doesn't matter that old plugins work. People won't be using them anyway.

I also think he did as good a job as he could under the circumstances. Despite that, some plugin authors ignored the warning. It's an unfortunate fact of life that plugins that don't comply with the API policies may well break.

-- MeredithLesly - 05 Jun 2006

Please remember that at the EdinburghReleaseMeeting2006x05x08 we agreed to this business rule:

If you change the API in a non-backwards-compatible way you also have to update all the broken plugins that have been actively developed within the last two years.

See irc log starting with [23:37] <Lavr>

-- PeterThoeny - 05 Jun 2006

I think "agreed to" is stretching things. And Kenneth's suggestion should really have said "plugins that comply with policy", although I assume that's what he meant.

But other than deprecating functions where necessary, I don't see that this is likely to be a problem. It is rather a shame that CDot was prevented from deprecating readFile=and =writeFile themselves with release 4, but since they're only supposed to be used within plugin work areas or outside of the TWiki hierarchy, I don't see that actually being much of a problem. (Unless, of course, you're expecting him to fix TagMePlugin, but that's between you and CDot.)

-- MeredithLesly - 05 Jun 2006

The following part is refactored out of TagMePluginDev. This is a more appropriate place I think.

-- PeterThoeny - 05 Jun 2006

Writing into the pub hierarchy is not approved API. I've had to remove the TagMePlugin from my installation because of that.

-- MeredithLesly - 24 May 2006 (comment in TagMePluginAppraisal)

You are entitled to your opinion but I am not sure what you mean by "not approved API". The official doc (TWiki03.TWikiPlugins#RecommendedStorageOfPluginData and TWiki04.TWikiPlugins#RecommendedStorageOfPluginData) recommends to store topic-specific data such as generated images in the topics attachment area.

-- PeterThoeny - 25 May 2006

I asked CDot yesterday about the use of pub before posting my comment about it. As of Dakar, nothing should be in pub except attachments. Topic-specific data should be saved via the TWiki::Func::saveAttachment().

So, no, it's not just my opinion: it's in the documentation.

-- MeredithLesly - 25 May 2006

The TagMePlugin is performance sensitive. If I change it to use the official attachment API it would be considerably slower because of the API and version control overhead. I do not see a reason to change the Plugin until we have a workable storable backend with production ready backends other than the RCS ones. Also, this Plugin supports Cairo and Dakar.

-- PeterThoeny - 25 May 2006

Given how important TWiki's reputation is when it comes to reliability, any plugin that violates API should be documentd as doing so. How else is an evaluator to decide whether to use a plugin? Citing performance issues for violating API is not an acceptable coding choice unless the plugin documentation clearly states that it does not stick to the documented API.

I'm not looking forward to going through rest of the plugins that I've installed looking for the same problems. And many TWiki administrators don't have the technical expertise to do that and so should be given the information in the documentation.

-- MeredithLesly - 25 May 2006

While I understand your motivation for a lightweight plugin, it would give a bad example to all plugin writers if we wouldn't restrict ourselves to official plugin API. And because of your position you would (will) make this plugin's behaviour also non-breakable (because of compatibility we would never be able to change it), setting a new coding standard which is not discussed and agreed upon.

I would like you to consider support from the coding community to make this plugin lightweight and compatible. I am sure there is enough knowledge available to improve this popular plugin.

-- ArthurClemens - 25 May 2006

Thanks Arthur for your constructive comment. It is possible to make the TagMePlugin perform also on Dakar by using conditional code: On Cairo, do the TWiki03.TWikiPlugins#RecommendedStorageOfPluginData in the pub area as documented, and on Dakar do the TWiki04.TWikiPlugins#RecommendedStorageOfPluginData in the Plugin work area as documented.

-- PeterThoeny - 25 May 2006

I would very much like to use the TagMePlugin, as I think it would be very useful in my current project, and it was only with reluctance that I removed it.

There was a conversation about the current implementation of TagMePlugin yesterday. There's no point in copying the whole conversation, since it's more easily read in the log file, but per the goals stated in BackToCodev, a link to the convo seems appropriate.

I was hoping to use it in my current project and am very disappointed that I can't. I don't know if I'll have time to implement a plugin with similar functionality and boy do I not want to have to. I share Arthur's concern that it sets a bad example when TWiki developers -- especially Peter, given his prominence in the wiki community -- don't comply with the Dakar PluginsApiPolicies, but my concern about best practices has probably overshadowed the desire to use it.

-- MeredithLesly - 04 Jun 2006

Would you care to state the reasons for why you can't use this plugin in your current project, so we can understand your concern better (only rename problem or others)?

I believe the reasons for chosen architecture with this plugin are very clear. No policy without an exception.

-- SteffenPoulsen - 04 Jun 2006

Some more facts:

The TagMePlugin is designed to work with good performance on Cairo and Dakar; it is designed so that it is possible to easily switch to a database if scaling is needed to over 100K topics (which would not be possible if the saveAttachment() / readAttachment() approach is used)
For Cairo, it complies 100% with the Plugin API and recommended way of storing Plugin specific data
For Dakar, it complies 100% with the Plugin API (but does not store Plugin specific data in the newly created workarea as suggested)

-- PeterThoeny - 04 Jun 2006

It's quite simple: tags are lost if a topic is renamed. I'm not inclined to use plugins that violate API policy -- I've removed a few for that reason -- but if it weren't for the rename problem I might have made an exception.

Thanks for clearing this up. -- SteffenPoulsen - 04 Jun 2006

I am not, however, about to tell my users that they can't rename topics, so that effectively precludes using TagMePlugin. It also raises the question of whether twiki.org should be using the plugin. I'm surprised that Peter's stated target audience -- large corporate intranets -- prohibit the renaming or moving of topics but then again I don't know much about large corporate practices.

I didn't see this stated anywhere, could you point to it (the fact that they prohibit renaming)? The way I see it, there's a fine "Limitations" section in the plugin topic; intranet admins are free to make an informed decision? -- SteffenPoulsen - 04 Jun 2006
If, of course, they notice it. I certainly was unaware of this limitation until very recently as, I suspect, were (or are) most of the TWiki folk. As evidence of this, using tagging for navigation purposes on twiki.org has been discussed quite a bit, but not a single person noted that doing this would mean that topics couldn't be renamed or moved. (Of course, Peter knew about the limitation and for whatever reason chose not to mention the implications of using tagging for navigation.) -- MeredithLesly - 04 Jun 2006
Please elaborate: How does the fact that tags aren't moved automatically lead to the fact(?) that topics can't be renamed or moved? -- SteffenPoulsen - 05 Jun 2006
Obviously it's not literally true that topics can't be renamed if TagMePlugin is used. But the value of tagging is greatly diminished. People are unlikely to spend time tagging topics if they know that the tags may disappear at any time, and using tags to find topics or to navigate within a site will be unreliable. Therefore, in practice it doesn't make much sense to use TagMePlugin and also have renaming enabled. -- MeredithLesly - 05 Jun 2006
If you plan to ignore the tags when you are renaming a topic I guess you are correct. -- SteffenPoulsen - 05 Jun 2006
Requiring people to write down a topic's tags and then restore the tags after a rename or move seems like an excessive burden. This is the sort of thing that people expect software to do automatically. And even if a person is willing to spend the time to restore the tags, they can't restore the state of the tags because:
- the tags of that topic now "belong" to that user (which messes up "my tags")
- it resets the "vote count" for that topic to 1 (messing up the stated reason for tag voting: "Tag voting is important for search ranking in large wikis" -- MeredithLesly - 05 Jun 2006
- I agree these are good factors to weigh in when doing the informed decision (but hardly prohibitive). -- SteffenPoulsen - 05 Jun 2006

I never said that it didn't comply with the Dakar API, by which I assume Peter means TWiki::Func: I actually don't know one way or the other. As I said above, however, it does not comply with the PluginsApiPolicies nor the deprecated usage of readFile and writeFile, which are important to ensure that TWiki is both reliable and yet able to not remain mired down in ancient architectural decisions.

Are these deprecated now, Func.pm has no notice of this, nor TWikiFuncDotPm? -- SteffenPoulsen - 04 Jun 2006

Crawford was blocked from deprecating the two functions for Dakar, so the best he could do was to document them so as to discourage them from being used on files in data or pub areas (that is, the TWiki content repository". In brief, the functions themselves are not deprecated (for the preceding reason), but their use is, except in a plugin's work area or outside of the TWiki content repository.

So they are not deprecated yet. Thanks for clearing this up. -- SteffenPoulsen - 05 Jun 2006
They're not, although CDot felt that they should have been. Most uses of the functions are deprecated, however, which seems to elude many people. This is one reason that it would have been good if CDot had been allowed to deprecate the functions themselves: it would have clarified the behaviour required to ensure plugins working in future versions. -- MeredithLesly - 05 Jun 2006

Any plugins that manipulate files within the TWiki content repository are liable to break in future versions. (This is a direct quote from the plugins api policy itself.) Unfortunately, this plugin is at risk of just that, which was my original point.

I don't see anything more to discuss. Peter, like any other plugin writer, is allowed to do whatever he wants with his plugin. They can support whatever versions they choose to. They can fix bugs or not. They can do their best to ensure that their plugins will work in future versions or not. And site designers can look at the decisions and decide whether or not they are able or willing to use a plugin.

As someone who is designing a site and would have liked to use TagMePlugin, I'm disappointed. As someone who is aware that Peter isn't just another plugin writer, I'm concerned at the example it sets, but I and others have already made that point. In the end, as long as Peter doesn't impose his plugin design decisions on twiki.org or the codebase -- that is, as long as he keeps his plugin-writer hat separate from his administrator or TWiki decision-maker hats -- it doesn't matter beyond what I've already said. In the end, it's just another TWiki plugin that will live or die on its merits.

-- MeredithLesly - 04 Jun 2006

Thanks. Agreed, WabiSabi still lives smile

-- SteffenPoulsen - 05 Jun 2006

It occures to me that Steffen, Meredith, Arthur and I spent considerable time on this one point (editing the TagMePlugin topic, the appraisal topic and chatting on irc.) And with a finger pointing. And with no measurable result.

Imagine a different scenario:

Problem is stated once
Someone creates the PluginHandlerForContentMove in the develop branch, and creates a patch for TWiki 4.0.2
I enhance the TagMePlugin with the rename handler, and move the data to the work area

Problem solved (would be by now.)

Now, which scenario is more efficient? Which one is more beneficial to the community? Which one builds up the community spirit?

-- PeterThoeny - 05 Jun 2006

It's quite simple: The policy for 4.0.x release is "only bug fixes". Thus there will be no new handler until 4.1 (at the earliest).

An alternative scenario:

Problem is stated once
You fix the plugin to comply with PluginsApiPolicies
Problem solved long before now, without discord

As a feeble silverlining, at least the deprecation issues regarding readFile and writeFile as well as the general principle about writing into the TWiki DB without using API functions have been clarified.

-- MeredithLesly - 05 Jun 2006

Discussion of Kenneth's proposed business rule

PTh inserted the following into PluginsApiPolicies: "If a developer changes the API in a non-backwards-compatible way he/she also has to update all the broken Plugins that have been actively developed within the last two years.:

As this was not agreed to by the community, merely tossed out by Kenneth, I have moved it from there to here. It's also completely unenforceable, and thus doesn't belong in an official policy statement.

Isn't there a pattern here: No matter how you feel about something, the community is always in agreement? -- SteffenPoulsen - 05 Jun 2006
I wouldn't go that far. But I talk to many of the inner community a lot, as well as having a lot of experience in developing software, so it's not a surprise that there's agreement with a lot of my positions. And it's not like it takes a computer scientist to realise that putting unforceable guarantees into a policy makes no sense.-- MeredithLesly - 06 Jun 2006
That's better, I already have a much easier time accepting "many of the inner community" than "the community". -- SteffenPoulsen - 06 Jun 2006
Sorry. I thought that was implied, since only the people who attend the TWiki meetings have the opportunity to vote (or whatever) on decisions. Clearly no one can speak for (or to!) the entire TWiki community. -- MeredithLesly - 06 Jun 2006

-- MeredithLesly - 05 Jun 2006

So you want to revisit this. If you read the log carefully you see this:

[23:41] <PeterThoeny> the more i think of what kenneth proposed, the more i like it
[23:41] <PeterThoeny> shall we introduce this business rule?
[23:41] <SamHasler> PeterThoeny>   kenneth: this does not address the issue of all those in-house built plugins
[23:42] <Soronthar> but it's a headstart
[23:42] <PeterThoeny> it is
[23:42] <Drusilla> Look. No one has any intentions of changing existing API in a non-compatible way.
[23:42] <Drusilla> Period.
[23:42] <Lavr> But it has happened.
[23:42] <SteffenPoulsen> I like it as a business rule as well
[23:43] <PeterThoeny> any objections?
[23:43] <Drusilla> Could we move to the next item please?
[23:43] <Soronthar>  Plugin API policy?
[23:43] <PeterThoeny> if not we can move on
[23:43] <Drusilla> Yes

-- PeterThoeny - 05 Jun 2006

No, I don't especially want to revisit it. I was desperate to move on at the time, and I sure don't want to discuss it again. (I also don't have any intention of changing the API in a incompatible way, so I'm not arguing with the goal.) But you can't promise plugin authors that their plugins will be fixed, since it's completely unenforceable. The point of document is to lay out what we're (for some very loose definition of "we') promising them in exchange for complying with the policies.

Haste makes waste :-/ -- SteffenPoulsen - 05 Jun 2006
And foolish stubbornness makes bad decisions. But it's not my reputation on the line. -- MeredithLesly - 05 Jun 2006

-- MeredithLesly - 05 Jun 2006

I was looking at the irc log for the above decision and found only a very short and rather disjointed discussion and nothing that would constitute an agreement of even those present, much less of the development team. This appears to me a pretty serious requirement that should not be shoved through. (I stand corrected having looked at the exempt above. My apologies for not reading more carefully the first time. Still, this appears to be, at best, a very "weak" consensus on which to base a potentially contentious policy. - LB)

Also, could someone point me to where the decision was made that plugins authors should be required (or strongly encouraged) to make plugins work across multiple versions. This seems to me an unusual requirement. Plugins for every portal or web app I've worked with always listed compatibility with versions and there was never any presumption that you could do a major upgrade and expect all your older plugins to work. At best, they offered a compatibility report and maybe disabled incompatible plugins.

-- LynnwoodBrown - 05 Jun 2006

Basically, the multiple-version plugin thing is Peter's and afaik, no decision was made about that either (AFAIK it wasn't even discussed). I'm already on record (above and elsewhere) as saying it's generally a bad idea.

-- MeredithLesly - 05 Jun 2006

If a developer changes the API in a non-backwards-compatible way he/she also has to update all the broken Plugins that have been actively developed within the last two years

I also was not party to any discussion regarding this statemement, and personally find it unacceptable. I do not understand why it was felt to be necessary. The deprecation policy was perfectly clear without it. It is unreasonable to make open-ended, unenforceable demands like that on core developers, especially when the code quality of many plugins make them virtually impossible to support. And what on earth does "actively developed" mean?

Of course developers will seek to support plugins authors - and I would hope vice-versa - but imposing ultimata is not the way to go. I have accordingly reworded the statement to something that I think most people should find acceptable. -- CrawfordCurrie - 05 Jun 2006

Argh, doing the Solomon-improvisation again are we? smile Nah, seems ok to me, practise and ideals need to meet somewhere - a stated best-effort intention is ok by me, though I liked the "guarantee" towards plugin developers better.

-- SteffenPoulsen - 06 Jun 2006

Two hours into a meeting isn't the best time to start a policy discussion, especially one that has not previously been discussed in a Codev topic first, and a lack of objections at that late hour shouldn't be misconstrued as agreement. I certainly didn't and don't agree to it; Crawford doesn't agree with it. He may have been away for the four minutes that Peter and Kenneth talked about it, but he's had time since then to read the meeting notes. No one who was not present, such as Sven, had a chance to express an opinion.

-- MeredithLesly - 08 Jun 2006

Please do not remove the grayed out content until we re-visit this rule (however unenforcable/undesirable you may find it).

-- PeterThoeny - 09 Jun 2006

It would be more respectful of the TWiki developers -- and, for that matter, plugin authors -- if proposed policy changes were discussed in a Codev topic and then, if necessary, discussed at a meeting with the item on the agenda. We all have opinions, frequently strong ones, but they belong in discussion topics, not policy ones.

-- MeredithLesly - 09 Jun 2006

Hey, a real edit war, cool! smile

I don't get the "No one who was not present had a chance to express an opinion"-part. Everybody has a chance to express their opinion on agenda minutes; I believe there's even a comment box on each and every meeting topic to make this an easy task for those unable to join in real time for our meetings.

Any suggestions on how to better avoid this suppressing of opinions (not giving them a chance) on the agenda items would be greatly appreciated.

At present it looks to me like one half of the participants leave our meetings thinking we had an agreement and the other half doesn't speak up; would be great to have this behaviour parked somehow.

-- SteffenPoulsen - 09 Jun 2006

This bit wasn't on the agenda, and was raised two hours into the meeting. Those who weren't falling asleep or playing solitaire were either too tired to discuss it, didn't take it seriously, or were desperate to move to the next agenda item (or all three). IIRC, there was no follow up in the next meeting as is usually the case when a matter is considered resolved. I'm also rather puzzled to find this particular bit popping up in a policy document a month after the meeting occurred. At any rate, to allow people to. comment on the proposal,

I'm glad you find edit wars cool. Personally, I find them annoying. It also tends to put rather a damper on "improving the documentation".

-- MeredithLesly - 09 Jun 2006

Let me make it clear that my proposal was not just "tossed out" or "unserious". I made a brief and clear proposal and I stand by it. And it makes a lot of sense and it is easy to enforce.

Core developers - have deprecated functions from the API and would probably have been deprecating more if someone had not stopped them. And surely their own plugins gets fixed quickly. And then they expect all other plugin authors to run to the computer and change their plugins. But this is not how the real world looks like. Plugin authors come and go. Think about our customers. This about our users that depend on those plugins. The same core developers are pushing the plugin authors hard to stick to the API to ensure that the plugin works in future. Well how can I be sure that a plugin I install will work next time I upgrade TWiki or next time again, when core developers deprecate API functions at their own will following a the current not very restrictive deprecation policy.

I stand by my proposal. Here are some arguments to think about.

The core developer that deprecates a function in the API normally replaces it with something better. He often updates his own plugin(s) at the same time. It is easy for him at this stage to implement the same changes to the other plugins.
The core developer will need to get an overview of what he destroys before he even suggests the deprecation.
The core developer will have a much easier argument when he deprecates a function. If you fix all the plugins - why argue against it? When the compatibility argument is resolved and the replacement API function is smarter/better/cleaner/safer/faster/whatever changing the API will actually be more effective than just following semi-religious dogmatic policies.
The current deprecation policy assumes plugins are all maintained. They are not! Many are NOT maintained but they are working! But quite many are still used by many happy users. It is the users that are the losers in the deprecation game.
The plugin authors are in many cases much less skilled in the art of programming perl and does not know the insides of TWiki. Many core developers assume that all developers have the same skillset as themselves. Last time I fixed a plugin it took 3 nights and a full weekend before it worked. It was 20-30 lines of code in total. If someone deprecates one of the API functions used in that plugin, it would probably take me several days to fix it again. For a core developer changing the API it will probably take 3 minutes because he known exactly what he just changed and how to replace the deprecated function.
It will encourage plugin developers to publish their plugins on TWiki.org instead of keeping them secret, because part of this deprecation policy will defacto be: if you do not share, we do not care.

What did I mean with "actively developed"? Use your common sense.

It is working with the current shipping TWiki.
Someone have updated the plugin within the last two years
People are still posting patches on the dev page.
A plugin that worked on Beijing but not in Cairo is clearly abbandonned.
A plugin that worked in Cairo but not in TWiki4 is in a gray area. Use your common sense. Maybe just replace the function in the SVN sources and leave it to someone else to later get the rest of the code working. Or post the code sniplet that is needed to replace the deprecated function on the dev page of the plugin. It does not have to take many minutes to do that. If you do not have these minutes - do not deprecate the function.

I cannot understand why this thing can end up with such a debate. To me - this is so obvious - when you just try to be a little bit customer focussed.

-- KennethLavrsen - 10 Jun 2006

I'd much prefer it if you created the proposal yourself, in a way that you consider fair. The only reason I created the topic at all was that you haven't been around and I was trying to restore the normal process of these things. The reason comments were out of order is that text was inserted into the policies page and that was what people were responding to.

The way the process generally works is that someone makes a proposal in a separate Codev topic, it's discussed there, and then if necessary put on the agenda at a meeting to be discussed there. This helps to avoid the kind of contentiousness that has occurred here. Burying a proposal at the end of a long discussion of many things isn't the best way to resolve things. In the end, it's up to you whether you choose follow the usual process or not.

The bottom line -- whatever the merits of the reasoning are -- is that this rule is unenforceable and thus should not be put into a policy topic.

This is not a company where developers can be put on probation or moved to another project or fired. This is an OSS project where people volunteer their time. I realise that this is a hard concept for some people to fully grasp, but what might be an appropriate business rule may well not make sense in an OSS context.

Crawford already responded to your concerns by putting in as strong a statement as can be made in an OSS project. I don't understand why that isn't sufficient.

-- MeredithLesly - 10 Jun 2006

The buttom of the buttom line is that Meredith is again manipulative and trying to split the TWiki Community trying to set us up against each other. Attacking Peter when his plugins "violates" her own definitions of what is the policy. But not at all attacking those plugins that other core developers have made that uses non-API functions in great numbers (which they have their good reasons to do because you cannot do everything in API and they are good at keeping their plugins up to date anyway - like Peter is doing)

Re-factoring is not only OK. It is the wiki way. But it has to be done properly!

-- KennethLavrsen - 10 Jun 2006

I didn't attack Peter, even if he chose to take it that way. I merely noted that his plugin violated API policy, which CDot later confirmed. Pointing out that a plugin is violating policy is not an attack. I am unable to use the plugin because of limitations that are due to how he implemented it. He has a right, just like any other plugin author, to choose to follow the API policies or not. I'm disappointed, however, because I had hoped to be able to use it and will probably not be have enough time to write a version without those limitations. If it turns out that any plugins that I am using have similar problems, I will either not use them or rewrite them so that they do comply.

I don't know which other plugins you are referring to, so they are presumably ones I am not using nor considered using. Had I done so, I would have made the same comment about them that I did about TagMePlugin. And, in fact, I've managed to get Sven angry at me in #twiki on more than one occasion when he suggested using non-API functionality to someone else, because I commented that I considered it a bad practice to do, and worse to suggest that others do it.

I also practice what I preach. None of my plugins reach into the core nor use non-API functions. I don't even use readFile and writeFile in the way that hasn't been deprecated. If there is functionality that is missing from the current API, I add a function to either FuncUsersContrib or MoreFuncContrib, which I then call in my plugin. If those contribs are not maintained by the core developers, then it is true that my plugins will break. By putting the functions into well-known places, however, I don't require anyone to read the code of the plugins; if the functions are maintained, which I expect they will be, my plugins will continue to work without change. I'm treating the core developers with respect by not forcing them to examine the code of my plugins or expecting special treatment. I'm treating the users of my plugins with respect by doing everything I can to ensure they will continue to work without change. I don't consider these goals "dividing the community"; I consider them an attempt to make the product more professional and reliable.

-- MeredithLesly - 10 Jun 2006

Forgive me to express my opinion openly, but I am with Kenneth on this one. A grep MeredithL log200605.txt log200606.txt | grep TagMePlugin | grep save | wc returns 46 topics saves on the TagMePlugin* topics done by Meredith. You can substract two counts for helping add related topic links (thanks Meredith), but besides that it is up to the reader to conclude what that those edits meant to be. Meredith please let me know how I can help you help building up the community spirit in a positive way.

-- PeterThoeny - 10 Jun 2006

Perhaps you might also note the number of those edits that were attempts to tone things down and/or make them more accurate? Of course, the reader might be better served by looking at the history of the topics rather than jumping to conclusions based on meaningless raw numbers.

-- MeredithLesly - 10 Jun 2006

Or to try to make the same point many times? Since you are asking, the appraisal diffs and the dev topic diffs show more details than the number of topic saves.

My question on how to help was sincere, not sarcastic. A community spirit where people like to help each other is very important to me.

Assuming you are indeed sincere, as you say, I have a couple of suggestions:
- Try to make sure there is a genuine consensus before changing things in important topics such as policy documents. Otherwise, it gives the appearance (whether accurate or not) that you are making decisions single-handedly which I, for better or worse, tend to take umbrage at. For example, it seemed that everyone was reasonably happy with the change that Crawford made in PluginsApiPolicies and I, for one, was surprised to see Kenneth's "business rule" inserted a month after the rather lackluster discussion. A lot of sturm und drang could have been avoided had this change been discussed more thoroughly first.
- Try not to take everything I say that may related to you as a personal attack. TagMePlugin was "singled out" because it was one of very few non-standard plugins I was thinking of using, and so I examined the code more closely than I would have otherwise. I would have said the same things about the Dakar API issues and the unfortunate (to me) limitation of not being able to rename topics no matter who had written it.
- Try to notice the many ways I've contributed to TWiki. It appears, although of course I may be wrong, that all you see is what offends you and not the considerable work that I've done. -- MeredithLesly - 12 Jun 2006

-- PeterThoeny - 11 Jun 2006

Meredith, you are about to close down communication between you and Peter again. And if that happens that means I'm probably back to watching your "Peter is not speaking to me"-comments several times a day again (which I really don't appreciate that much) :-/

Do you think there's any chance you could try to use the open channel for common good? I.e.:

If you are about to nag/being a PITA: Consider doing something else instead?
If you are about to contribute, but not in a constructive way: Consider doing something else instead?
If you are about to start / engage in a personal attack: Consider doing something else instead?
If you are about to contribute in a way that brings TWiki from a) to b): Let it all flow!

Being assertive about everything is a usable approach in a lot of situations; spicing it up with a constructive and forward-facing, actioning angle makes it so much more interesting (imho).

-- SteffenPoulsen - 11 Jun 2006

No worries, as I rarely participate #twiki any more. I prefer to stick to fixing bugs, working on develop, and having productive out-of-channel discussions.

Worked for the Romans; can probably be applied here also with good effect (divide and conqueror). Thanks! -- SteffenPoulsen - 11 Jun 2006

-- MeredithLesly - 11 Jun 2006

I think there is a lot of unnecessary heat in this discussion. Let's just inject a sense of realism, here, and review some facts:

While plugins functions have been deprecated, none have been removed - even prior to the PluginsApiPolicies being established.
I personally put a lot of work into porting plugins to Dakar; mainly by removing their dependencies on unpubished APIs. So did Will, and others. Steffen has put in even more work on the more marginal contributions. "Core developers" do not just walk away from plugins.
The PluginsApiPolicies require 2 major releases before an API can be removed after deprecation. Since TWiki releases tend to appear once a year, this means 2 years.
Core developers come and go, same as plugins developers. You say "if you do not share, we do not care". But who is we?
Many plugins are marked "Contact author first". So, the deprecator is forced to contact every such authors, some of whom don't respond for up to a year.
Most plugins ship without any tests, so you have no way of verifying if they work.
Many plugins require external technologies that core coders may not have access to (interfaces to other systems).
Many plugins are so badly coded (alphabetti spaghetti) that it is unreasonable to expect another coder to modify them.
There are over 200 plugins, of which perhaps only a fraction are used on more than one site.

BasicForm
TopicClassification	TWikiDevDoc
TopicSummary
InterestedParties
RelatedTopics	TWikiPlugins, Plugins.ReadmeFirst, TWikiMetaDotPm, FunctionsNeededForPlugins

Plugins API Policy discussions

Discussion on common code base

Brief discussion on other API functionality

Back to code base issues

Discussion on deprecating certain usage of readFile and writeFile

Discussion of Kenneth's proposed business rule

Discussion on deprecating certain usage of `readFile` and `writeFile`