[Wikipedia-l] Waiting for 1.0

Sat Jul 17 13:53:24 UTC 2004

On Sat, Jul 17, 2004 at 01:48:09PM +0100, Neil Harris wrote:
> I disagree. I believe that most users would prefer the ability to
> express shades of opinion. A numerical rating allows more information to
> be gathered; users who want to assert an all-or-nothing opinion can
> still vote 0 or 10. I would then suggest feeding the results to a robust
> estimator (using the median would be a good first hack) to prodice a
> fine-grained estimate. It would also be useful to have a robust spread
> measure to detect contentious articles.
> 
> By having a wide range of fine-grained estimates, and perhaps adding in
> factors for linkage, we can then prune the encyclopedia to any desired
> size by adjusting the threshold for inclusion.

1. There are serious problems with having more than 2 options,
   but on the other hand 2 options are not enough to express all opinions.
   What is the meaning of each number ? Everyone knows when to give worst
   and best available rating, but what kind of article deserves 4 or 7 ?
   It works quite well on IMDB, where people select ratings thinking
   "this movie was about as good as movie X, which got 7.8, so i'll rate it 8".
   Will the ratings of the articles get coherent enough to provide such guildlines ?
   Or are we going to attach descriptions to each rating, in which case we
   get to another problem ...

2. How to produce a single number from all the ratings.
   There's no natural linearity in ratings, and if we were to assign descriptions
   to each rating, what's the "average" of "bad", "bad", "great" and "needs minor
   improvements" ?
   Median is conceptually better than mean, as it only requires ratings
   to be comparable ("bad" < "needs minor improvements" < "good" < "great" etc.),
   but it's quite dissatisfying that it completely ignores the difference
   between possible ratings below the median (ratings "horrible" and
   "needs minor improvements" are equivalent if the average is
   "needs minor improvements" or better), and above it
   ("great" == "needs minor improvements", if median is "needs minor improvements"
   or worse).

3. If we were to implement rating scheme, I think it would be the best option
   to have different each user's rating options and summary ratings.
   The summary ratings could be based on median, variance of ratings,
   and the number of votes.

   For example:
   Available rating options: {horrible, needs major improvements, needs minor improvements,
   good, outstanding}
   Possible summary ratings: {
	not rated (no votes, or few votes and high variance),
	controversial (many votes, too high variance)
	extremely controversial (many votes, ratings dominated by horrible and outstanding)
	horrible (many votes, median 'horrible', small variance),
	probably horrible (median 'horrible', few votes or moderate variance),
	needs major improvements,
	probably needs major improvement,
	...
   }