10/4/05

Helpful?

So I mentioned in a blog post a while back that I thought the "Helpful" button might be broken, and may not be the most accurate way to gauge a RateItAll member's on site reputation.

So why do I think this? Let's look for example at the page devoted to Cartoon Network. Scroll down the page a bit a take a look at the post that Magellan left. His comment, "Great!" received two helpful votes.

Is this really a helpful comment? Or did this comment receive two helpful votes based on the fact that he's been on the site a long time, knows a bunch of folks, has been known to leave helpful comments elsewhere, has a "moderator" tag, etc., etc. All kinds of possible reasons that this comment was considered helpful - unfortunately, the comment itself, was probably not one of them.

On a larger scale, this sort of use of the helpful button can be deceptive. For example, there is a Top 25 RIA user (no longer active with the site) who received nearly 40% of his/her helpful votes from a single user.

Is this cheating? No, of course not. There are no rules as to what constitutes a helpful vote. Is this misleading? Perhaps, when you consider that the purpose of Top XX tags is to indicate to readers of the site whose opinions have earned a little more respect among the community. To continue with this thought, if the reader knew that 40% of the helpful votes resulting in that Top 25 tag came from a single user, perhaps this would cause them to think twice about considering that reviewer one of the elite members of the site.

So how can we get around this sort of issue? How can we come up with a measure of site reputation that alerts newcomers to the site of a reviewer's reputation, as gauged by the community?

There's no easy answer to this. One of the things that I've been looking at is a consideration of all the different factors that might go into what might a "trusted" reviewer look like, from a purely statistical point of view. Things like tenure on the site, number of helpful votes, distribution of helpful votes, number of folks trusting that member, reputation of those trust that member, who the member trusts, etc, etc.

My goal is to come up with a formula that converts all of these helpful type behaviors into some sort of a tag to help readers know who has earned a reputation on RateItAll.

That's not to say that a newcomer to the site who has only left a single post cannot be trusted. Of course not. All we would be doing with this tag is to say that statistically speaking, you're better off trusting someone who has demonstrated a bunch of these behaviors than someone who hasn't.

My guess is that everyone who is reading this blog has earned a TREMENDOUS amount of reputation here. The point of this is not to separate folks into tiers - but to provide some quantifiable measure of how to recognize those who have helped make RIA the community what it is.

We're still a ways away from this. It's actually pretty complicated math to come up with something that scales, so it's a fairly big project.

However, in the next couple of days, we should be rolling out a summary table that tracks things like reviews left, new items suggested, weblist items managed, incoming trust votes, etc. The purpose of this table will be to recognize folks who have contributed to the community on a variety of different fronts - not just by receiving helpful votes.

Stay tuned for more info....

2 comments:

  1. Knowing how my own helpfuls were generated it is a safe bet that I am "that" reviewer alluded to in this post. Heh. I am not unique though. The vast majority of helpful votes have never been given out based on anything other than personal like for the reviewer or the comment. That transformed into a more generalized "buddy" system currently running rampant on RIA.

    Previously, raters came and posted and knew those who agreed with them or those who liked them would click helpful before long and in quantity. Now however, with the buddy system, they come at the same time or if a "buddy" is posting at the current time they repost their comments so it can get those "well deserved helpfuls". Irishgit is a big offender of that. Although he would never admit to being part of the buddy network. He does routinely repost his comments en masse when various reviewers are actively posting or a topic is hot.

    The overall idea of quantifying trust is still futile. Many in the business world may know what KPI are. And KPI are used often to determine who stays or goes in business. It isn't pretty either. While KPI "TRY" to measure effort the result, all too often, is that cold hard numbers get someone the axe who busted their ass to do well and perform their job adequately. But then again it is cold hard numbers.

    And when it comes to the internet the ONLY thing you have ultimately would be some form of cold hard numbers. By its very nature it is impersonal and the ability to be anonymous, remain anonymous, or even "pretend" who you really are, is quite easy.

    In fact there are numerous sites, surverys, and written material covering just how untrustworthy the internet is.

    Specifically to RIA... it is far too easy for users to just fake it. In private the vast majority of users talk down about the others or otherwise admit dislike for various behaviour, attitude and posts. But in public they try very hard to "maintain" appearences.

    I picked on Numbah once but the fact is few take him seriously and many do in fact view him as a buffoon. Will those users say it publicly and to his face? Get real. The key concern most often are those helpfuls. It is something to consider for site users.

    How many people are giving you BS? The fact is the site admins "pride" themselves on the sites protection of user identities and the "anonymous nature" of it all. For purposes of running such a site it amy be a necessary evil. But it is an evil. No real names are used. No genuine attitudes or personalities can be presented. You just have to take their word for it. RIA even bans users from giving out real names and the like. At least in some repsects regarding posts.

    But it is an evil. Such as with "open" governement. Would the government be MORE or LESS trusted if the votes and meetings were all anonymous? Ah... And that is why it is a negative and not a positive.

    When it comes to trust the first thing you must have is openness. There is absolutely no openness on the site and users a anonymous and specificall KEPT as anonymous.

    While many may think they know each other the vast majority admit they know very little and what is presented in private or in public is rarely convincing in regard to trust.

    That is why such lists as "Which RIA reviewer would you have over for dinner" make so little sense. In my time on the site I was exactly who I am. It's why I never felt the need to apologize and why I pissed so many people off.

    It is a nice idea to think that this user or that user is a buddy but the fact is if the users of RIA were actually GENUINE, actually SINCERE they would argue much more often and would rarely stroke egos.

    But helpfuls get in the way so instead of various users posting on numbah sincerely that as a user:

    "He has a sense of humor sometimes but most of his posts are short, stupid, and uninteresting. 3 Stars."

    They post something much less honest or trustworthy such as:

    "Numbah makes me laugh. 5 Stars!".

    Or with irishgit they might actually be honest and say:

    "He is a fence-sitting, cynic, who takes his own comments as far more worthwhile than they actually are".

    But instead most say "Irishgit is a tough irish guy who I admire and find fascinating."

    And those are just examples. All of those comments could be viewed as accurate. My point is that on RIA there is no "spectrum" of commentary. Especially reagrding other users. They are either too scared to speak negatively in public or fear the loss of helpfuls.

    And in real life we are much less accomodating. Most people don't praise or get along with most people around them. They vertainly don't go out of their way to stroke the egos of so many that they actually dislike on many levels. But the site offers helpfuls and there is no way to actually gauge trust.

    There may come a day when some cold hard numbers are used but it will still be guess work and a "system". Any online gamer can tell you that a system can be broken and circumvented. At te very least it can be manipulated. And the fact everyone is specifically anonymous means manipulation will be much easier than on a secure game server.

    The site itself will steal user works as well. So if the site itself can't be trusted to show some measure of integrity how can its users? And how can the users be seen as more than prostitutes of commentary when they admit the site uses them and shows situational ethics?

    And several users have admitted that. The one I'd guess gave me about 30-40% of my helpfuls admitted it and is still their. In private they are quite adamant that the site is not fair to users very often. Including them. But they choose to sacrifice some amount of integrity as they remain after such an admission.

    And most users make such a sacrifice when they acknowledge various abuse and lack of principles on the sites part but they remain anyway.

    Eventually the site will do away with helpfuls completely. Many users will likely leave when that happens. The "charm" will be gone. And once a few users realize that a "trust" system can be abused and maniuplated many more will likely leave.

    If the site actually wanted to gain back some lost integrity the first step would be eliminating the helpfuls. The bigger and more effective step would be doing away with all lists on other users. One last step forward would be elimination of political or religious lists.

    I DEFY anyone to read over the popular political lists and try to contend the commentary is factual OR fair. It sure as hell isn't "honest". 90% of it is party line BS. So the general conservative would never give way on the "abortion" issue nor would the general liberal give way on the "death penalty" issue.

    And the same can be said of religious lists. But those lists will remain despite the fact that everyone on gods green earth knows what b*llsh*t it all is. And the reason comes down to hits and revenue.

    A single political post, controversial or tame, will garner helpfuls or unhelpfuls by at least a 10 to 1 margin over the typical movie post. All while being mostly party line BS or general stubborness.

    So there are methods available to actually regain some integrity and put forth a site that focuses on USEFUL opinions and commentary. It is much easier to "trust" a persons view on a movie or a product. There is usually little strong bias or bad feelings. But in many topics such as religion or politics there is almost no trust to be had. Bias or harsh feelings aren't just abundant. They are most often the guiding force.

    Users are dumb sheep though. All that ultimately matters is revenue. That is why the site allows so much complete crap. Sex sells and when it comes to opinions, so does crap. And the sheep continue to dump loads of crap in the form of hits. Whether it is a crap comment or a crap list.

    ReplyDelete
  2. To be honest, I like the political and religious lists the best. I know the site it designed to gauge response to consumer products, but the users here are politically and religiously diverse enough to make these rating polls interesting indicators of public opinion.

    And I really don't pay attention to the most trusted buttons. All that means is that they left lots of comments. If the comment itself makes sense, then I find it helpful regardless of who left it.

    ReplyDelete

Apture