> 3. SOURCEFORGE PEER RATINGS SYSTEM
> We've implemented a peer rating system on SourceForge. This feature
> was inspired by the trust algorithms designed by Raph Levien for
> advogato.com. The system is a new implementation which contains a few
> twists. SourceForge users will be able to rate each other on a variety
> of aspects, including: teamwork, code, design, leadership and
> reliability.
>
> Who has the strongest code-fu? Who's the best project manager? Who's
> the best designer? Who's the most reliable? In the end, there can be
> only one.
>
> Users can start rating each other right away. However, because the
> system is based on trust, it will take a while for ratings votes to be
> tallied as "official" (your votes don't completely count until someone
> who's "trusted" has rated you). For more information, see your (or any
> other developer's) profile page, at:
> http://sourceforge.net/users/[developernamehere]

Here is an example of a rated user (a SourceForge sysadmin): http://sourceforge.net/users/precision

NicholasLee had the idea of using this algorithm for TWiki. Any thoughts?

-- PeterThoeny - 20 Sep 2000

Please elaborate on the purpose.

Do you mean implementing this feature in TWiki to rate twiki users or only for here on sourceforge for contributors?

If as a general rating system implemented in TWiki, how will it be used? I assume it will be optional cuz at least in corporate environments, people don't want to be rated needlessly when the whole purpose of TWiki is to enable collaboration, not competition.

-- ManpreetSingh - 20 Sep 2000

As a TWiki feature. Might be possible to use it to say, give write access to certain pages to certain people.

You can find some comments from the originator of the algorithm here:

• http://advogato.com/trust-metric.html
• http://advogato.com/mission.html
• http://www.cs.berkeley.edu/~raph/fc.ps

I am very interested in some kind of rating system for topics. I would like to use TWiki as a support tool for teaching and I want to encourage/discourage good/bad contributions.

I was thinking to implement/adapt a peer rating system like the Slashdot's one, but Raph Levien could be good also.

The rating can be used in several ways, depending on user preferences:

• to suggest the best topics
• to give positive/negative feedback to the participant, e.g. showing "your best topic is rated 74% and your worst 22%" in his/her personal page
• to block the visualization of spam topics

-- AndreaSterbini - 21 Sep 2000

There is a lot of criticism of Slashdot rating system all the time. People get power hungry. There are disruptive people who would rate someone down just for the fun(?) of it. Same thing happens with XP points on http://perlmonks.org. If implemented, I'd still want this to be totally disableable. I'd like to see collaboration rather than point competition. And this is probably one factor in Wiki's success also. You can say anything without fear of criticism. Isn't that already listed in advantages of Wiki/TWiki somewhere? At least in corporate world and general software industry, people would go mad if they were to get rated by their peers for nothing.

If you do decide to implement it, please make it optional to use.

-- ManpreetSingh - 21 Sep 2000

A Wiki system in corporate use is a totally open system that fosters sharing of ideas. I agree with Manpreet, a peer rating system does not really fit the pure Wiki idea, nor the corporate world. I can see a use in teaching, as Andrea pointed out. If someone implements it, it should go as a FeatureAddOn.

FWIW, TWiki.Codev is the place for brainstorming, so anything goes! We have some very good ideas here, and some of them turn out to be part of the distribution.

-- PeterThoeny - 21 Sep 2000

I agree with you. In a small community a peer rating system is negative or useless ... here the signal/noise ratio is very high because we are focused on the common interest of making the best Wiki we can think of.

In a different setting, e.g. when teaching a course to 200 students, I wonder if the teachers are able to cope with the possible presence of HostileUsers.

TWiki already gives us the safety-belt of RCS ... and of user authentication. If the guest account is disabled then no student can hide under anonimity. This is probably enough for discouraging misuse of a collaborative area between us teachers and the students.

A peer-rating-system is probably too rude to users/students ... If used to rate people it can probably produce distortions and reinforce the "power" of few ones.

An idea can be to rate contributions ... e.g. to use something like the "3 votes" system that is used at http://developer.java.sun.com to vote for the 3 most important bugs/features the user would like to see solved.

This is an example of a rating system that is useful because it gives a positive contribution to the system (in this case to the support staff at Sun).

What I am looking for is a rating system that can help both the teacher and the student for his/her coopeartive learning process.

Is there somebody here that has noticed peer-rating-systems that give valuable information to the users? If yes, please, drop a line to me.

-- AndreaSterbini - 22 Sep 2000

Well it looks like I may be implementing this for DrKW - as there is some respect for "intellectual stamina" hereabouts, it seems that we invite criticism - positive or negative. So with apologies to those who would like to see a system with an inbuilt constructive bias...

Current feature proposal is (broadly):

• Store review data in MySQL (with DBI interface)
• Rate quality of topic 1-2-3-4-5
• Rate relevance of topic 1-2-3-4-5
• Constain primary comments with a predefined drop box
• Free text field
• Tie review to revision
• Notify reviewer on change (optional - possibly not in first rev)
• Display topic stats on view
• Add reviewer stats to personal pages

This seems to be enough for now - in the spirit of XP I would like to get something working and see how/if it is accepted.

-- SteveRoe - 01 Jun 2001

Great idea. One of the qualities that people were saying about .Plastic was the fact that you could rate peoples contributions. Something like this in TWiki would be great.

-- MartinCleaver - 01 Jun 2001

Please, add an option:

• the number of votes a user can post can be limited (like in Sun's developer site) so that all people is equally powerful ...
  • the limit could be specified through a web preference

-- AndreaSterbini - 21 Jun 2001

Hi! - I did consider giving each person a fixed set of points to be somehow spread around. Having discussed this within DrKW, we felt that an active reviewer who made the effort to review many pages should not be disincentivized (?) by diluting or limiting their contribution.

There were also some practical considerations in the design which made this more complex to write...

• Need to read individual totals on an add
• Need to provide a way for reviewers to delete stale reviews to free up points (right now - for simplicity - reviews are like wiki topics - they cannot be deleted except by the sys admin)

Certainly these are things that could be added - so when I'm done with the first cut, you should be able to take a look and we can maybe identify a list of "rev 2" improvements...

-- SteveRoe - 25 Jun 2001

Another thing for general consideration is the storage of review data. When looking at the data associated with a review:

• Reviewer
• Topic url
• Topic rev
• Date
• Rating
• Comment

And the ways that reviews need to be assembled and processed: SELECT topic, ORDER BY date, INSERT INTO, AGGREGATE functions, it seems that a database would be a very natural storage medium. So the current plan is to use DBI/DBD/MySQL to store these critters.

I am conscious that this is contrary to the lightweight "run anywhere" nature of TWiki - but it just didn't seem like good engineering to start writing a database...on the positive side, this is a plugin so that it does not make installation of a database mandatory.

-- SteveRoe - 25 Jun 2001

I have added some comments about the use of include to allow the reviewed topic to be included alongside the reviews on a single page. I am trying to include a feature that allows external web pages to be reviewed alongside wiki pages...see IncludeUrlVariable. Comments on the usefulness of this would be much appreciated.

-- SteveRoe - 29 Jun 2001

OK - the first revision of this is now available at PeerPlugin. Enjoy! Please continue to comment below on this topic.

-- SteveRoe - 22 Aug 2001

Wow. Well cool. Can´t wait for it to be installed at twiki.org ;^)

-- MartinCleaver - 22 Aug 2001

Just a thought - it is quite likely that this plugin could be made to work without installing a database - there is a file only DBD and it does not do anything that is MySQL dependent to my knowledge. Sooo...it may be possible to get this up and running without going through the pain of installing and maintaining a database (which I guess is a pain for the TWiki.org site???).

-- SteveRoe - 23 Aug 2001

Your plugin is GREAT! My students will love it!

(this means that I must add some minimal InternationalizationSupport in my home release....)

-- AndreaSterbini - 24 Aug 2001

Blush - thanks! However, I cannot take all the credit - the idea was raised by several of the DrKW IT team and driven through by JohnTalintyre.

-- SteveRoe - 24 Aug 2001

Has anyone (besides SourceForge) actually implemented this plug in for testing?

-- WiccanTN - M3R 22 Mar 2002

I've not tried this plugin, but it's on my (all too long) List Of Things To Do :->. Interested parties may wish to check out http://kuro5hin.org/, a discussion site that depends very heavily (and reasonably successfully) on its rating system. The Kuro5hin meta section also includes considerable discussion of the risks and advantages of both their rating system and rating systems in general.

-- NicMcPhee - 03 Aug 2002

Interesting discussion here. I'll have to dive in more soon. I feel rating systems in general will become more and more important as information overload increases. Different things can be rated too: topics, people, changes to topics, etc. These are the only real objects I can think that might be useful as targets of a rating system.

Shifting gears to focus on rating of topics, I can see relevance if applied to important topics but this is sort of covered by WebStatistics as far as what people look at, right? That's good, but a rating system would give a qualitatively different perspective. However implemented the UI is also very important. I look forward to trying out the PeerPlugin soon.

PeerPlugin seems to require MySQL. That's a pretty heavy requirement. Couldn't topic ratings be stored in metadata and just parsed out per topic?

-- GrantBow - 29 Dec 2002