Helpful?

So I mentioned in a blog post a while back that I thought the "Helpful" button might be broken, and may not be the most accurate way to gauge a RateItAll member's on site reputation.

So why do I think this? Let's look for example at the page devoted to Cartoon Network. Scroll down the page a bit a take a look at the post that Magellan left. His comment, "Great!" received two helpful votes.

Is this really a helpful comment? Or did this comment receive two helpful votes based on the fact that he's been on the site a long time, knows a bunch of folks, has been known to leave helpful comments elsewhere, has a "moderator" tag, etc., etc. All kinds of possible reasons that this comment was considered helpful - unfortunately, the comment itself, was probably not one of them.

On a larger scale, this sort of use of the helpful button can be deceptive. For example, there is a Top 25 RIA user (no longer active with the site) who received nearly 40% of his/her helpful votes from a single user.

Is this cheating? No, of course not. There are no rules as to what constitutes a helpful vote. Is this misleading? Perhaps, when you consider that the purpose of Top XX tags is to indicate to readers of the site whose opinions have earned a little more respect among the community. To continue with this thought, if the reader knew that 40% of the helpful votes resulting in that Top 25 tag came from a single user, perhaps this would cause them to think twice about considering that reviewer one of the elite members of the site.

So how can we get around this sort of issue? How can we come up with a measure of site reputation that alerts newcomers to the site of a reviewer's reputation, as gauged by the community?

There's no easy answer to this. One of the things that I've been looking at is a consideration of all the different factors that might go into what might a "trusted" reviewer look like, from a purely statistical point of view. Things like tenure on the site, number of helpful votes, distribution of helpful votes, number of folks trusting that member, reputation of those trust that member, who the member trusts, etc, etc.

My goal is to come up with a formula that converts all of these helpful type behaviors into some sort of a tag to help readers know who has earned a reputation on RateItAll.

That's not to say that a newcomer to the site who has only left a single post cannot be trusted. Of course not. All we would be doing with this tag is to say that statistically speaking, you're better off trusting someone who has demonstrated a bunch of these behaviors than someone who hasn't.

My guess is that everyone who is reading this blog has earned a TREMENDOUS amount of reputation here. The point of this is not to separate folks into tiers - but to provide some quantifiable measure of how to recognize those who have helped make RIA the community what it is.

We're still a ways away from this. It's actually pretty complicated math to come up with something that scales, so it's a fairly big project.

However, in the next couple of days, we should be rolling out a summary table that tracks things like reviews left, new items suggested, weblist items managed, incoming trust votes, etc. The purpose of this table will be to recognize folks who have contributed to the community on a variety of different fronts - not just by receiving helpful votes.

Stay tuned for more info....