Debrief

It’s been an awful week for our site.

In the past week alone, RateItAll has lost four excellent reviewers: IJR, StarkTruth, Ridgewalker, and Wiseguy. Each has chosen to leave the site, and exercised their rights as RateItAll members to permanently delete their accounts.

As I’ve seen a lot of rumors floating around as to the circumstances regarding their departures, I thought it was important that I clear the air and present the facts as I understand them.

As a rule, I try not to discuss site disciplinary matters publicly – these have historically been left between the admin and the member. But as these are extraordinary circumstances, I’m going to be as transparent as possible in order to present the most accurate picture of what transpired.

Those involved are welcome to comment on this blog if they feel that I got anything wrong.


IJR

As we all know, IJR was a longtime reviewer who was very popular. He was prolific, he was funny, and he was pretty unique. At the time of his departure, he was third ranked in the list of Most Helpful Reviewers.

About four weeks ago, I noticed a series of very odd sounding reviews from him. They were not written in his style, and had an oddly formal tone to them. I ran one of the reviews through Google, and found it be copied directly from another site. There was no citation given to the author of the review.

I messaged IJR, pointed him to the site rule about plagiarism, and asked him not do it again.

The following day it happened again, so I warned him again.

The following day it happened again, so I warned him again, and said that if this continued to happen, I’d have no choice but to suspend his account for two weeks.

Note: it’s almost unheard of for me to give three separate warnings, in rapid succession, for the same, fairly serious, violation. IJR was given this courtesy only because of his standing with the site.

A day or two later, it predictably happened again, so I suspended his account.

When the two week suspension was up, I reactivated his account. Within 24 hours or so, IJR had sent messages to a few folks (myself included), complaining of an old boys’ network, and deleted his account.

I feel that my hands were tied in this particular situation. The violations were so blatant and so defiant, that my choices were either to say, “screw it, the rules don’t apply to IJR.” Or to take action. I guess I could have asked him if there was something else that was bothering him.

Looking back, I wish I had.

My personal opinion is that IJR was intent, bordering on obsessed, with becoming the reviewer with the most helpful votes. When he started to feel like it wasn’t going to happen, he prepared himself to self destruct by flaunting the rules. I wouldn’t be surprised if he felt that the odds were unfairly stacked against him, hence the “old boys network” (which I take to mean cronyism) accusations.


StarkTruth and Ridgewalker

As most of you probably know, StarkTruth and Ridgewalker are a couple. Whip smart, Stark is an excellent writer who made a pretty significant impact on the site with both her reviews and her lists. From an admin perspective, I considered Stark to be very much a “tough cookie.” While she didn’t start fights, she had a history of coming in pretty hard when she felt that she was slighted. I had a bit of an admin run in with her shortly after she joined, when she began messaging a number of regular reviewers with the false information that I was planning on selling the site. A heads up was given to me by several regulars. I’m not sure why she thought this (I’ve never actively tried to sell the site), or why she felt the need to try and start a panic.

Despite her impressive contributions, I never got the feeling she particularly liked (or trusted) the site (or me).

And then there’s Ridge. In the history of RateItAll, I can’t remember anybody who has been more universally liked and admired than Ridgewalker. Besides writing the best reviews and making the best lists (and helping others with their lists), he was also the first to welcome new users and the first to try and talk overheated reviewers down. He was a giant on RateItAll, and somebody who I personally admire a great deal. When he fell ill, I sent him a care package – in the site’s nearly 8 years online, this is the only time I’ve done this.

About two months ago, Ridgewalker brought it to my attention that a certain RateItAll reviewer was accumulating helpfuls at an unnatural rate – something like 2,000 in a 24 hour period. I investigated the matter and found this to indeed be the case.

What was strange is that the helpfuls were coming from a user that was not logged in, who was at the same IP address of reviewer in question. In other words, somebody seemed to be logging out, and giving helpfuls to themself.

As far as I knew, this was an impossible scenario. You can’t give helpful votes if you’re not logged in, and you certainly can’t give helpful votes to yourself while logged in.

This was happening at around the same time that RateItAll was going to a multi server configuration. A known bug at this time is that folks were getting logged out in mid browse, as they bounced back and forth between servers. In other words, some members found themselves in a weird limbo status where they were logged in at one moment, and logged out the next. Then logged in again.

In this weird limbo status, it was apparently possible to give your own reviews helpful votes.

This is important, because it gets to intent. This was not a situation in which the member had gone out of their way to set up a duplicate account with the sole purpose of giving themselves helpful votes. My understanding is that this was a situation when a member was browsing the site, and suddenly realized that they had the ability to give themselves helpful votes.

So, I suspended the user’s account for 3-4 days while I sorted out the technical glitch that enabled this, and deleted the bogus helpful votes.

Amazingly, we discovered that at that moment, it was not possible to subtract helpful votes in our system, as explained here. I felt that it was important enough to get this particular user’s bogus helpfuls off the books, that I commissioned the development work to fix this. When this was fixed, not only did the offending user lose their bogus helpful votes, but everybody’s tallies were recalibrated to reflect subtracted helpful votes (primarily from folks who had left the site).

I also formally warned the user, and expressed my disappointment at the bad judgment.

So why didn’t I ban this person? For a couple of reasons. First, the person is a regular, and regulars nearly always get the benefit of the doubt. Secondly, there have been at least 7 other cases of members in the top 100 who have been discovered showing bad judgment in regards to manipulating the system (specifically, setting up dupe accounts) in order to pad their own tallies. All of these reviewers were warned, and given a second chance. None of them have broken the rules again in regards to dupe accounts.

Finally, given that the vote getting was enabled by a technical glitch, and not premeditated, I don’t see how it would have been appropriate to jump the warning and suspension stages of discipline, and go right to ban. Just as there’s a difference between “stealing” a bag of money that falls out of the back of a truck and robbing a bank, there’s a difference between creating a false ID to give yourself helpfuls, and suddenly realizing that you have the power to give yourself helpfuls.

There was bad judgment displayed here, but it simply was not a ban worthy violation. If anything, it was less severe than stuff that I had let go with a warning previously. Banning this user would have been utterly inconsistent with how I’ve managed similar situations in the past.

So here’s where things get murky.

At exactly the same time as this helpful vote situation, Ridge told me that he and the member described above, had been emailing with each other. Allegedly, there was some inappropriate stuff emailed (not messaged through RIA) to Ridge. This allegedly inappropriate stuff came to Stark’s attention, who understandably, was not happy about the situation.

Stark posted an angry personal attack against the reviewer who had been emailing with Ridge on RateItAll (though without actually naming the person), and the reviewer responded with a threatening message. Unfortunately, both of these posts were not immediately noticed, and stayed on the site for longer than I would have liked.

Eventually, the two posts were deleted, and both reviewers were warned. No disciplinary action was taken against Stark (as it was an extenuating circumstance) or the other reviewer (as it was a response to an attack.)

And of course, it is not within RateItAll’s domain to discipline RIA members for what (allegedly) happens on their own time, via their own personal email servers. From the perspective of site discipline, any email or real world correspondence that happens between RIA members is simply not relevant.

I did make at least one critical mistake in my handling of this situation – I discussed some of the details of the bogus vote getting with Ridge. Over the years, I discussed A LOT of site issues with Ridge. Both strategic, and tactical. In a lot of ways, Ridge was one of my most trusted advisors on things related to RIA. He was probably the most respected member of the RIA community, he had the site’s best interest in mind, and he is an incredibly smart, mature, and insightful guy. Of course I’d want to tap his wisdom.

So why was it a mistake to discuss the bogus helpfuls with Ridge? Because of the timing, and his relationship with Stark.

I can imagine Stark’s outrage when she found out that not only was a RateItAll member emailing inappropriate stuff to her man, but that same member was accumulating bogus helpful votes.

Flash forward to last Friday. Stark again posted a personal attack on the reviewer (without specifically naming the person). I removed it. Stark immediately posted it again. I removed it and suspended her account to try and talk things with Stark via Ridge.

There was no way that I was going to allow a public, nasty, personal shootout on the site.

In messaging with Ridge, it became pretty clear that Stark (and probably Ridge) expected the reviewer to be banned, and saw my decision not to ban the user as a slap in the face. I responded that it was not within my domain to ban somebody for something that happened away from RateItAll. Ridge responded that he understood that, but what about the bogus helpfuls? I responded that the user had already gone through a suspension, and based on how I’ve moderated the site in the past, I could not justify going right to a ban for that particular violation.

So we were stuck. Stark was furious that an RIA member could get away with inflating their helpfuls and crossing the line with her boyfriend, and she wasn’t even given the opportunity to respond on the site. Ridge, already concerned about what he perceived as a growing problem with RIA’s culture (more on this to follow), was also in the awkward situation in which his continued participation on RIA might be seen as disrespectful to Stark (my speculation).

What a mess, what a mess, what a mess.

Soon after, I received a note from Ridge requesting that I reactivate Stark’s account so that she could self delete. I complied. Stark deleted her account. Soon after, Ridge deleted his account. This was a devastating surprise, though looking back, he probably felt himself with no other options.

At a micro level, I feel that my only mistake was discussing the details of the bogus votes with Ridge – you don’t tell something to one half of a couple, and expect it not to get to the other half. That’s not fair. This information, which was incomplete and should have been confidential, only inflamed the situation even more.

I stand by my decision not to ban the other reviewer, and not to let my respect and admiration for Ridge make me moderate the site in an unfair manner.

But at a macro level, I failed miserably. The site lost its best reviewer and list maker and friend in Ridge. Worse, I don’t really think he wanted to leave (I could be wrong about that).

My gut tells me that Stark was going to take off regardless, but perhaps more skilled diplomacy by me could have salvaged her RIA career as well.

Wiseguy

This brings us to Saturday.

Wiseguy was a respected and passionate member of RateItAll. His main interest was politics, which meant that he got into it from time to time with folks (including me), but I think that there was a lot of respect for him on both sides of the political aisle. He’s a guy that I know cared deeply about the site, and when he said something regarding site management issues, I listened.

Saturday morning, Wiseguy posted the following video to YouTube expressing concern about the recent loss of talent from RateItAll. He also suggested that “if it had been him, he’d be gone” (I’m assuming that he meant the bogus helpfuls) and that I needed to get a handle on some of the shenanigans that certain reviewers were getting away with. Finally, he suggested that I work harder to keep the talent from leaving.



His departure came as a shock to me. I had sensed some growing frustration from him over recent months – specifically, that he felt that liberals were treated better on RIA than conservatives. But still, I didn’t expect him to leave, and am pretty bummed out about it.

Two things in particular make me sad about Guy’s departure; 1) that it wasn’t his fight; and 2) he was given bad / incomplete information. The rumor that was floating around was that a reviewer had hacked into RateItAll and given themselves thousands of helpful votes, and was allowed to stay. This is very different scenario than what really happened – a technical glitch on RateItAll allowed a reviewer to give themself helpful votes.

So no Guy. You would not have been banned if it were you. Nobody would have, except perhaps somebody on final warning status. And you’re more than welcome to come back – we’d be thrilled to have you.


Moving Forward

So where are we at? Well, we’re bending, but hopefully not breaking. I’m heartened by some of the level headed things that folks are saying here. There’s a lot of talent, and humor, and class left on RateItAll, and although we’ve taken a terrible hit this week and lost some great, great people, I’m hopeful that we can get things moving in the right direction again.

One of the things that I’d like to address in the next few days is this issue of “culture.” I’m prioritizing community things over all else for now – step one is going to be getting rid of that damn not helpful button. No need for the negativity. I also have some other ideas for some tools with which we can recognize our friends, but without over dependence on the helpful button. Finally, given that helpful votes have become the de facto currency of the site, I believe that we are in need of better regulation around abusing the helpful button, so that we can make excessive cronyism unprofitable so to speak. Stay tuned.

RateItAll Has An Intern!

I'd like everyone to extend a warm welcome to Jose. Jose is a grad student at UC Hayward focused on e-business. He's an exchange student from Granada, Spain.

His primary role with RIA is going to be making sure that we have coverage on the site for things that are hot in the news. He's also going to be helping with responding to user feedback, and rolling out our new feedback widget. He may also blog occasionally.

I met Jose on the basketball court - he's the best European player that I've ever played with / against, and was at one point, getting paid to play hoops.

So please be nice to our new intern, and if you get a chance, shoot him a note to say hi.

New Mail Server Online

Wow. So far, it seems like the new mail server is having a substantial impact on overall site performance, specifically in terms of:

- Leaving a review (page is no longer hanging as email alerts are sent out)
- Sending a personal message
- Inviting others to rate / join

I knew our old mail server was crappy, but I didn't know it was THAT crappy.

Search Engine Just Got A Lot Better

We just made a small but important tweak to our search engine.

As you know, the site is partitioned into three primary channels: RIA, Weblists, and Local. Until just a few minutes ago, our general search results were prioritized in that order. So if you did a search for George W. Bush, you would see results from RIA before you saw results from Weblists - we did this because we could vouch for the quality of the listing on RIA, whereas the quality of the listings on Weblists run the gamut from excellent to weak.

The problem with the way things were set up, is that we didn't have control over the weighting involved in driving the results. So the search engine would display ALL results from RIA regarding George W. Bush, even if it was just a casual mention of the word "Bush" in passing in a review, ahead of any listing from Weblists - even ahead of a listing that was specifically about George W. Bush. This was really annoying, and it made it really hard to find, say, where Barry Bonds ranked in the Biggest Jerks in Sports list.

This is no longer the case. We now have control over the weighting for the three partitions for the site. Currently, a search for George W. Bush will still show results from RIA first, but a Weblist listing for George W. Bush will now outrank a passing reference of "Bush" in a review in the RIA channel.

This is a small tweak, but I think you'll find it improves the quality of the search engine dramatically.

I Love Rate-O-Matic

Rate-O-Matic is one of my favorite RIA features to come down the pipe in a long time. I especially like the "speed rate" option for specific lists.

There's a few things I really like about this feature:

- It helps you find stuff that you wouldn't have otherwise rated
- Your rating is not biased by what other people have said
- It's a quick and easy way to blast through a list
- It's a quick and easy way to blast through your own new weblist, to announce its presence to the site

According to our stats package, in just a few short days, the Rate-O-Matic page has become the second most popular page on the site (after Recent Comments).

It seems that I'm not the only person enjoying this feature, and over the next few days you'll see more integration points where you can hook into this service.

Email Alerts Turned Off

RateItAll's mail server is a piece of garbage. It's always crapping out on us, and worse, it slows down the reviewing process on the site - here's why:

When somebody sets up an email alert on an item or for a reviewer, when a review is posted, the site waits for the email alert to be generated before loading the page. Because the mail server is so crappy, this sometimes takes a few seconds (sometimes more than a few), meaning the page hangs until that connection is made with the mail server.

Well the server has been getting progressively worse. Today, it was taking so long to post a review, that I've decided to just turn email alerts off until we can get a new mail server in place.

So, we're shelling out the $600 for a new server, which should be up sometime tomorrow. Until then, there will be no email alerts (new messages, new ratings, etc.) sent from RateItAll.

The good news is that the reviewing process is super fast now.

RateItAll Under Attack

RIA is currently under a mild DDOS (Dedicated Denial of Service Attack) from somewhere in China. Basically what this means is that zombie computers are being directed to tie up as much of our bandwidth as possible, in an attempt to cripple the site.

I think we have things under control, as there appears to have been only a handful of IP addresses involved. We're filtering these out as they pop up, and things seem to be stabilizing.

New Tool

Hi all, in case you missed it, there's now a new tool live on the site that lets you delete the comments that you post under a specific review.

This is consistent with RIA's core philosophy to give its members complete control over the opinions and content that they post to the site. I'm not aware of another site that provides tools so that members can post their content on other sites, shares ad revenue, and allows its members full deletion functionality.

Couple of Profile Page Tweaks

You may have noticed a small change to the profile page:

We've moved the member's historical site activity up on the page, so it's the first thing you see after the person's "About Me" section. I'm finding it nice to not have to scroll way the heck down to read someone's review history.

Also, we're experimenting with caching some of the live stats on the profile page, so that they update once a day. This is to try and speed up the load time - I'm still not happy with the speed of these pages.

Changing Your Profile Image

Some of you have reported an error message when trying to switch your profile image. This is now fixed.

Image Syncing Now Live

We now have images synchronized across all of our web servers. No more "now you see it, now you don't."

Please let me know if you see anything screwy when adding images to profile pages, weblists, or item pages.

Now it's on to the fun stuff! Rate-O-Matic comes next :)

New Web Server Online

Our new web server is now online - this means we now have three web servers and a db server sharing the site's traffic load.

I don't expect the new server to make the site faster, but I do expect it to help significantly in making sure that big traffic surges don't cripple the site. In sum, RateItAll is now better equipped to handle significant traffic bursts.

So, in the past week, we have:

- optimized the Recent Comments page
- optimized the profile pages
- added a new web server

This is pretty boring stuff, but critically important.

We're also very, very close to rolling in the image server fix. Stay tuned.

Profile Pages

OK, we made some optimizations to the profile pages and we now have the heaviest profile pages (with lots of widgets, rating history, etc) down to about 5-6 seconds, from (gulp) 15 seconds plus.

This is a big improvement, but still not fast enough. We'll keep plugging away.

Just Bought Another Server

We'll hopefully have it in the mix some time next week. We're also, very close to putting in that image fix so that images are synced across all servers. I'm also hopeful that we will have the profile pages optimized shortly for faster page loads.

This is stuff is boring from a development perspective, but a fast site makes the RIA experience much better IMHO.

Record Traffic Day

Yesterday was an all time record traffic day for us. Depending on what stats package you believe, we did anywhere from 41K to 49K total visits.

For the most part, the site held up, though it got a little bit sluggish from time to time.

For the moment, we have prioritized site performance issues over everything else - some cool features have been put on the backburner until we can get some problem pages (e.g. profiles) loading fast.

I've also requested a quote for another web server.

Recent Comments Page Breakthrough

Guys, I think we've solved the chronic slow loading of the recent comments page from a logged in state. The solution has been right front of our eyes - divide the load into small chunks. Enjoy. Now I can go back to hitting that page about 700 times per day.

Weekly Goals

Our goals for the week:

1) Fix the image syncing issue
2) Speed up the recent comments page for logged in users
3) Speed up the profile pages

Images

Some of you may have noticed strange stuff going on with profile and item images. Sometimes they're there, sometimes they're not.

This has to do with when we went to a mulitple web server configuration a few weeks ago. Because the images are saved locally on either one server or the other, you will see or them or not see them depending on what server you're hitting.

We have a fix in the works that will sync all images across all servers - this should be up some time next week.

Couple of Small Fixes Went Live

We pushed a few very minor fixes live last night:

- The default display in the "Reviews from Trusted Network" display went from two degrees away, to one degree away. In other words, you'll now only see reviews from folks in your immediate trusted network on your profile page.

- We changed the Overall Rating display on each item page from stars to numbers. I was getting tired of accidentally clicking the cumulative rating stars when I was trying to post my rating.

- We fixed it so clicking on the "comments" link associated with any review will automatically scroll you down the page to where the comments begin.

RateItAll on the BBC

Over the weekend, the BBC included RateItAll in a segment called "The New Wave of Silicon Valley Start-Ups." If you watch the video clip, RateItAll's mention is about 8 minutes in. It's a brief mention, but very positive nonetheless.