Debrief

Late yesterday afternoon, we had our best web server fail. The traffic load was too much for the remaining two web servers to handle, so performance tanked. Our host was unable to troubleshoot the issue, and we were told that they would take another shot at it Saturday (today) AM.

So, at this point, we knew a couple of things; we knew that our remaining two servers were not up to our traffic load, and that there was no guarantee that we would be able to get our big server back online today.

So yesterday evening, Automatt and I made the decision to try and migrate the service to what is called a "cloud computing" hosting provider. This is a new type of host that gives you virtual machines, and the ability to quickly add capacity as your traffic grows. It's all self serve, so it's conceivable that we would be able to get back up and running late Friday night.

So we configured five virtual webservers and a database server, migrated our database over to the new host (which took a few hours as it's pretty big), and pointed the "RateItAll.com"domain to the new IP address. By the time everything was ready to go, Automatt and I had worked through the night and it was after 6AM PST. Unfortunately, when you update DNS information, there's not an immediate update across the Web. In our case, some folks were not able to access the site until about 1:30PM today.

Once we had RateItAll up an running on the new service provider and everybody around the world was able to hit the site, it became quickly apparent that our new virtual host did not have the computing power to handle RateItAll's load. Many more virtual servers would have been needed, and it was unclear that even then, these virtual servers would be able to handle the dynamic nature of RateItAll.

Around the same time, our original host got back to us and let us know that they had managed to repair our big web server that had gone down late on Friday.

So at this point, early this afternoon, we made the decision to go back to our original hosting configuration, which we were sure could handle RateItAll's traffic. Some of you may still be seeing an IP address as opposed to the RateItAll.com domain in the web address line - this should be resolved back to RateItAll.com shortly.

Once we pointed the domain name back to our original IP address, for whatever the reason, bots from Google and Yahoo started hammering us at much higher intensity than normal. This has impacted performance today, making the site very slow. We are now working to slow down these crawlers to get the site speed back up.

It's been a brutal weekend so far with very little sleep and a lot of stress. The good news is that we learned some valuable info - mainly, that cloud computing (at least the provider we tried) is still too early in the game to be a viable host for a big traffic site like RateItAll. We also learned that although we have multiple servers, we are very much dependent on a single server to maintain our quality of service - this will be changing this week.

Thanks for your patience as we go through this.

Caution: Server Maintenance in Process

Guys, we are working on the site. Tread carefully rating stuff... site performance will be dicey for a while.

I'll post a full debrief shortly of the last 18 hours - it's been an interesting experiment.

Site Coming Back

Site should be available to all over the next few minutes. Thanks to Automatt for the long night of work put in to get us live.

Status

We are making progress, but expect it will be another hour before we are back online.

Update

Automatt and I decided not to wait to see what happens with our lead web server tomorrow - we're moving over to another service provider tonight, so that we can have a snappy website through the weekend.

We expect to back online by 1AM PST.  More update to follow.

Hardware Failure

Our biggest, baddest web server fell off the grid a few hours ago, and is not responding.

So, we are currently running on two smaller web servers, which are pretty much pegged under our regular load.

We'll know more tomorrow AM about whether this box is recoverable - best case scenario is that it's just a power supply issue, which would be resolvable.

Expect performance to be sluggish until we get this box (or a replacement) online.

Ads in the Recent Reviews River

You'll notice that we've moved ads on the recent reviews page from the right sidebar into the river. We're trying out different stuff to try and find the balance between what converts, and what isn't uber annoying.

Commenting Now Working Across MySpace

You can now access comments on your reviews from your MySpace account. Here's a screenshot. This also means that folks using RIA from MySpace will receive your comments that you post on their review on RIA. With great power comes great responsibility :)

Best Error Page Ever

Not that we intend to serve up many error pages, but in the off chance that we do.... we now have the best error page ever.

Try clicking this link - and make sure to keep refreshing (the photo will change).

More RIA Trip Reporting

There's nothing I like better than a good RIA trip report... Zuchinibut has just set one up for his recent trip to Oregon.  Has anyone else been to these places?  Chime in!

Battling Some Application Errors

We're on it.... stay tuned.

Messaging Bug Fixed

I'm sure everyone has noticed that annoying little bug in which RateItAll personal message threads reverse the names of who is saying what.

For example, when people send me hate messages, the thread might read something like this.


AngryUser69: I'm sorry you feel that way. We do the best we can.

Lawrence: RateItAll Sucks! My pet monkey could do a better job moderating this site!

When in fact, the exchange was really like this.

Lawrence: I'm sorry you feel that way. We do the best we can.

AngryUser69: RateItAll Sucks! My pet monkey could do a better job moderating this site!

For obvious reasons, this could get confusing. I'm happy to say that it's fixed now.

MySpace Invites Now Working

For those of you using RIA for MySpace, the invite tool is now working. You can challenge people to take a test, you can share a review, or you can share the app itself.

The only catch is that you have to invite one at a time.... you click on the name of the person that you want to share / invite with, then click on the next one after the box pops up.

The invitation shows up in the recipient's bulletin space.

Remixed RIA Video from Hyve Up

My friends over at Hyve Up have created a cool video with background techno music of me trying to explain what RIA is:

You can rate HyveUp with this widget:

Technology Blogs on RateItAll

RateItAll Funding

Hey All, just a heads up that RateItAll has raised some funding.

The good news is that not much is going to change, other than than our ability to provide a better service to the RIA community. A few more developers and an awesome designer will be joining Automatt and myself. This will mean they we will be able to fix things and add features at a must faster rate, and that things should work better.

We remain committed to our core philosophy of giving our members more rights than just about any other site - owning your reviews, sharing revenue on your reviews, sharing your reviews on places like MySpace and Facebook, and of course, leaving whenever you want and taking all of your stuff with you.

Thanks for your support.

Irishgit's SF Tour

I caught up with Irishgit tonight, live and in person in San Francisco. We hit Zeitgeist, Arinell's for pizza, Casanova, and then Arinell's for pizza again. We gossiped about RIA stuff and talked a lot of sports and politics.

Overall it was a pretty good night.

IG is the third person I've met through RIA. Previously I met JonTheMan and Minkey. It's amazing how well you can get to know somebody by reading what they think about everything.

Rebooting DB Late Tonight

Guys, we are doing some scheduled maintenance late this evening that will required a DB reboot, bringing the site down for a 15-30 minutes.

Automatt, will you comment with the exact time?

Speaking of Trip Reports...

Speaking of using RIA to document trip reports, check out Irishgit's reviews of the places and things that he's seen while driving down the Pacific Coast.

Word has it he will be hitting San Francisco's legendary Zeitgeist beer garden tomorrow at 7PM. I'm in as well - any other local RIA'rs are welcome to stop by.

Obama and Fear

OK, we have a couple of new quizzes live on MS and Facebook. An Obama VP quiz based on GTH's Obama VP Candidate list, and a fear test based on Jamie McBain's list of things you are afraid of.

And to my conservative friends out there, please don't read anything into the juxtaposition of the "Fear" and "Obama" quizzes.

I can't believe we don't have a 2008 Democratic candidate list yet

Who wants to take it on?

Here's the 2008 Republican VP candidate list.

Trip Reports via RateItAll

One of my favorite things to do after taking a trip is to write up the things and places that I saw. I just wrote up my trip to Atlanta for my brother's bachelors party. We went to a ball game, and I reviewed Turner Field, a couple of Atlanta bars, and the hotel I stayed at.

For me, RateItAll is almost a diary of sorts, and this is one my favorite things about the site.

Two New Quizzes

I added a Job Compatibility Test and a Break Up Quiz to MySpace and Facebook. Check them out!

Also, thanks for all of the good suggestions for more compatibility quizzes... I think we need a way for anybody to convert their lists into a FB / MS quiz.

Agree is the New Funny

Sometimes when I'm utterly baffled by a review, but I want to acknowledge that I read it, I will drop an "agree" vote on it.

Often this makes no sense:

Reviewer: "I remember growing up in Kentucky snorting the Grizz four time per day."

Me: Agree

I didn't grow up in Kentucky, and I certainly have never ingested the Grizz. Yet I somehow feel compelled to drop an "agree" vote.

Is there any cure for what I have?

Posting a Video Review: A How To from Genghis the Hun

GenghisTheHun has very helpfully put together a video explaining how to post a video review to RateItAll.

Genghis chose to shoot the clip in sort of a Blair Witch style - minimal light, with only a hand and several eerie lights visible. I found myself overcome with an inexplicable feeling of dread after watching it.

But if you can get past the horror film techniques, his tutorial is quite helpful.

Here are the cliff notes:

- Record your video using a web cam or camera
- Export it to your desktop
- Upload it to YouTube
- Copy the embed code from YouTube and paste it into the review field

But I encourage you to watch the whole video. Thanks GTH!:

Suggestions for New Compatibility Tests?

If anybody has suggestions for lists that could be converted into compatibility tests, please let me know. These tests are a great way to drive traffic to specific lists... if you have a list that you think might work as a compatibility test (it should be mainstream, easily rate-able, and with pictures and images), please let me know.