Debrief on Yesterday's Performance Issues

Pretty much all day yesterday there were performance problems with the site related to rating things, and adding new things to the database.

As this site is called Rate It All, these sorts of problems are pretty devastating.

Here's what happened. When unregistered reviewers rate something, we create a "state" that remembers their review so that when they go on to register, their review can post. These folks are stored in the database as "Anons." As you know, no review can go live on RateItAll until there is a registered user behind it. We have to do it this way, or the spammers would overwhelm us.

Well, it turns out that a lot of folks abandon ship once it gets time to join RateItAll.

We hadn't cleaned out the Anons in a little while, and there were 2.7M (yes, MILLION) Anon accounts that were jamming up our database.

This is why things were sluggish yesterday, and we're still working on cleaning these accounts out. Going forward, we'll create a process so that these abandoned Anon accounts are deleted if they do not complete registration...

I spoke too soon...

....we're still working on it.

I think we have things working again

Cross your fingers.

Man, this has been a long day... up at 4am and still working. My eyes hurt.

downtime imminent

Guys, our database is acting strangely. We're going to reboot it, and it may take another hour or so until things are running smoothly again.

I'm noticing a bunch of performance issues on the site

I'm having problems rating things, and submitting new item suggestions.

We're looking into it.

Speed Rating Just Got a Lot Better

I think that Speed Rating is one of the better developments on RateItAll in recent months. But one thing that really bugged me - if you clicked "skip" on an item that you didn't know, it would eventually feed you the item again.... sometimes almost immediately.

Well, we've fixed this. If you skip an item while speed rating, you will not see that item for the rest of your session.

This also applies to our compatibility tests on Facebook. Hopefully, this will eliminate reviews like this one.

BTW, if you haven't tried the compatibility tests on our Facebook app, you should... they're fun.

Be Careful Out There

Don't be fooled into thinking that the Internet is this parallel universe in which stuff we write and post doesn't have real life repercussions. Check out these guys who got arrested for their YouTube video.

It can feel sometimes that RateItAll is a small, insulated, private corner of the web. It's not. We get a lot of traffic, and a lot of people are reading what you post. Please be careful of saying things that you might regret, and that might get you in trouble in the real world.

Two Weblist Tweaks Went Live

We rolled in two new changes to how we manage Weblists today.

Change #1 - On RateItAll, we let people leave whenever they want, removing all of their content in the process. We firmly believe that your thoughts are your own, and if you want to leave - for any reason - you can do it, with no questions asked. We're pretty unique on the web that way.

However, one point of contention has been that when disgruntled reviews self purge they not only remove their own reviews, but by deleting weblists that they had published, they remove all of the reviews that others had posted within those lists. At times, this has bordered on the catastrophic. Hundreds if not thousands of other peoples' reviews deleted in one fell swoop.

Well we changed this today. You can still leave whenever you want, deleting all of your reviews, comments, helpful votes, profile, etc - but from now on, all public weblists and their associated reviews will be relocated to this account should the list's creator choose to leave. This will eliminate the scenario in which a disgruntled member has the power to delete the reviews of others.

Change #2 - As some of you may remember, we have a "User Page Only" status that can be manually applied by admins to Weblists that don't adhere to our guidelines. Typically, this status is only used on lists that are exact duplicates of lits that already exist on the site. This User Page Only hides the list from our main index, and prevents reviews posted from showing up on the recent comments page. Today, we've also hidden these lists from our search results. The idea is to avoid confusion for new users who may see numerous identical lists and listings in the search results.

Couple of Fixes Went Live Today

• We added the HOF link
• We darkened the link color a little more (yes, I know I said I wasn't going any darker)
• We've added "add description" and "add image" links to Rate-O-Matic / SpeedRate. So if you're speed rating, and an item doesn't have a description or a photo, you can add one. This sort of editing is really helpful for your fellow Weblisters (and RIA)
• We've added the Yahoo! shopping ads to the right side of the Speed Rate pages. This is one of those instances where I think ads actually make the experience better, because of the photos. For example, I was speed rating an actor list today, and it was helpful to see photos for the folks I couldn't identify from their names.
• We fixed a bug so that the text is now showing up besides list and item page images, as opposed to below
• We blended the AdSense ads into the site better (this actually happened two days ago)
• We fixed an annoying bug where none of our Drama Movies were showing up in the search results. Now they are.

Thanks to Jan and Peter for their hard work in getting these fixes out today. Next week we're making some fixes to our FB app and hopefully, rolling out a Bebo app.

* Oh Yeah, one more: It is again possible to post YouTube videos, and other Flash widgets, in reviews (as long as you're posting from the item page."

RateItAll Hall of Fame Link

We've added a HOF link to the main nav bar - you can see it right next to the "about us" link on every page.

This page tracks the top 100 all time reviewers on RateItAll in terms of helpful votes. We've also added columns that show total Funny votes, and total Agree votes.

Couple of things jump out at me:

- Numbah is running away with the funniest reviewer title.
- LadyJesusFan gets an impressive number of "agree" votes
- One of the folks with tons of reviews and not a single "funny" vote is my friend in real life Wiggum. Coincidentally, Wiggum is not very funny in real life either.

Expect to see some changes to the helpful rankings... I'm still working through the best way to deal with some of the helpful vote giving shenanigans that were going on a few months back. The best thing I've been able to come up with is to come up with a cap that represents the most helpful votes that an indvidual could have naturally, and in good faith given to another individuall by this date in time. All votes beyond that number coming from a single individual will be erased.

As always, I don't blame the receivers of unnaturally high levels of votes. It happens, and for the most part, it's out of the person's control. But I will request that for those of you who are determined to manipulate the helpful rankings, without taking into consideration the quality of the post..... please stop. It sort of breaks the game for everybody else.

I don't have a timetable on when this is going to happen.... but it is going to happen.

Who knew there were so many Dingo related advertisements?

Watchmouse: RateItAll Trounces Facebook in Terms of Performance

There was an interesting study recently published by a company called Watchmouse that looked at some of the biggest social networks and how they did in terms of site speed and staying up.

I'm happy to say that RateItAll trounced Facebook in terms of overall performance.

I'm less happy to say that of the 104 major social networks profiled, RateItAll is one of the slowest and one of the worst performing.

When RateItAll is sluggish, it's something that literally makes me sick to my stomach. If you ask RateItAll's developers, or RateItAll's host, or my wife - they'll all tell you.... I turn into a monster when the site is down, or when pages aren't loading as fast as they should, or when things lock up. It's such a basic thing - a slow loading site can ruin even the best features and content.

Over the past year or so, we've tried everything. We've brought in database experts. We bought a massive, state of the art, quad database. We've added load balancing across three web servers. We've removed features and functionality.

I think we're making progress, but we're still not there yet.

But trust me... making RateItAll faster is something that is almost always on my mind and we're going to keep plugging away at it.

Kids - Please Learn How to Program

There is nothing more frustrating in my job than not being able to help with hard core technical issues that arise. This morning, we had an issue with a table in our database. It cost us about three hours of uptime.

My role in getting this fixed? Firing off emails and texts to our lead developer and host saying stuff like "it's still timing out." Why? Because I didn't know how to go in and fix it myself.

It was an incredibly frustrating and helpless feeling.

Anyway, my personal motto is "don't be a victim." So, I'm checking out some programming classes at the local community college.

Our Database has Locked Up

.... rebooting now.

Argh.

New Item Suggestions Cleaned Out

I'm happy to report that due to the efforts of Jose the Super Intern, we are caught up on new item suggestions for the first time...... well, let's just say for the first time in a long time.

Here's a little refresher as to how we handle new item suggestions for lists:

For the RateItAll and Local lists, we respond to them as quickly as we can get to them.

For Weblists, we try and wait seven days to give the list maker a chance to make a go or no go decision on the suggestion. What this means is that if you suggest a new item for a Weblist - even if the Weblist's owner is not active on the site - it will get responded to in about 7-10 days.

And remember - if you suggest a new item for a list (RateItAll, Weblist, or Local) by clicking the link that says "add a new item to this list" one any list page, you get credit for that page in terms of revenue sharing. This is an extremely powerful way to "claim" important pages of the site that also helps us out a lot as well. For example, if there is a hot new cell phone coming out, why not submit a listing? When everybody wants one of these phones next Christmas, you might have some real earning potential.

Couple O' Tweaks

We made a couple of small tweaks last night:

- Based on repeated pestering from Genghis The Hun and others, we've darkened the orange link font a bit more. This is as dark as we're going to go dammit, so don't bug me anymore :)

- We've added a new RIA for Facebook icon in the top right corner of every page. Thanks Dan.

- We've added a bit more padding to the text column on the information pages like this one, to make them a bit easier to read

- We've added item descriptions to the right column of the SpeedRate /Rate-o-Matic pages to make it easier to identify the item that is being rated

Server Maintenance tonight at 1AM EST

Guys, we're going offline for about half an hour tonight. We've added more memory to the database and have to reboot with the new settings. Hopefully we'll see some nice performance bounce.

Closing out 2007

I hope that everyone is having a safe and joyful holiday season. I've been more or less unplugged for a few days, eating too much and spending time with family. It's been great.

But now it's time to get back to work.

You may have noticed that we've been featuring some 2007 lists on the home page - I think that we're just scratching the surface on what can be done here. If you have some ideas as to some themes that might make good lists capturing aspects of the year that was 2007, now's the time to push them out into the world. If they're good enough, we'll put them on the home page.

Name this movie

We get tons of feedback messages at RateItAll. TONS. Occasionally, we get requests like this:

You may help me with my search. I am now 63 years old and I saw a movie (drama?) in the second half of the 50´s or all along the 60´s but I do not remenber the name of the movie, the actors etc. The only thing I remember is part of the story. At a certain time of the movie one man is given as dead by a train accident. However the autorities were wrong because this man himself saw in a newspaper his name as the one who died. He did not say anything to the autorities and his family which thought he realy was dead. He never show up again to his family and start being a beggar. Afterwords the christmas season is coming and at the christmas eve he came to his house but did not knock at the door. One of the sons saw him through the window and came out asking to the beggar if he had no family and if he wants to join them inside, without recognize him as his father. This is what I remember from the movie. I understand it is very dificult to help me with only this description but could you try? It is a very emotional movie. I appreciate a lot.

Thank you

Can anybody help this person out? MA Duron?

Widgets and Apps

Most of the RIA regulars know by now that I'm pretty obsessed with widgets and the fragmentation of the web. I even write a blog devoted to widgets. In the world of RIA, this widget obsession means doing whatever I can to make sure that RIA content and functionality can be enjoyed not just on RateItAll.com, but all over the web - on social networking sites like Facebook, on blogs, and on any other site with a social component.

Over the past year or so, we've started putting together the tools to make this happen. I've taken a shot at summarizing all of the different tools available on RateItAll that allow you to enjoy RateItAll on other websites. Check out this summary of RateItAll widgets.