Showing posts with label performance. Show all posts
Showing posts with label performance. Show all posts
3/10/09
Scheduled Downtime: Tonight at 9PM PST
We will be in maintenance mode tonight from about 9PM until 10PM as we make some performance tweaks to our database.
8/5/08
Cutting Over to a New Host - DOWNTIME IMMINENT
Guys, as we continue to deal with site and rating engagement growth, we will be cutting over to a new host today.
Currently our architecture looks like this: one pretty good database server, one very fast web server, one OK web server, one weak web server.
As of later tonight, our architecture will look like this: One very good database server, five very fast web servers.
We expect a noticeable improvement in performance, and of course, the ability to handle several times the amount of traffic that we can today.
We're going to be down for about six hours while we transfer our massive database to the new host.
Currently our architecture looks like this: one pretty good database server, one very fast web server, one OK web server, one weak web server.
As of later tonight, our architecture will look like this: One very good database server, five very fast web servers.
We expect a noticeable improvement in performance, and of course, the ability to handle several times the amount of traffic that we can today.
We're going to be down for about six hours while we transfer our massive database to the new host.
6/27/08
Hardware Failure
Our biggest, baddest web server fell off the grid a few hours ago, and is not responding.
So, we are currently running on two smaller web servers, which are pretty much pegged under our regular load.
We'll know more tomorrow AM about whether this box is recoverable - best case scenario is that it's just a power supply issue, which would be resolvable.
Expect performance to be sluggish until we get this box (or a replacement) online.
So, we are currently running on two smaller web servers, which are pretty much pegged under our regular load.
We'll know more tomorrow AM about whether this box is recoverable - best case scenario is that it's just a power supply issue, which would be resolvable.
Expect performance to be sluggish until we get this box (or a replacement) online.
5/27/08
Hey University of Singapore - Knock it Off
In addition to the million or so humans a month that visit RateItAll, there are also thousands of "bots" - non human visitors who crawl and scrape our site for content.
Some of these bots are benign - for example Google, Yahoo, and MSN all operate bots / spiders that are used to index our content so that it can show up in their respective search engines. These bots follow strict rules, and respond to our requests to slow down if they are hitting us too hard. We love them.
But others are rogues. Pounding our servers without regards to our requests, and scraping our (yours and RateItAll's) content so that they can display it elsewhere and try and make money off of it.
Today, one of those rogue bots was hitting pretty hard - we tracked it back to the University of Singapore. They were hitting us about twice as aggressively as Google, which is pretty aggressive.
So if you guys over there in Singapore are reading this, nice work. You overdid it and now you are permanently banned.
Some of these bots are benign - for example Google, Yahoo, and MSN all operate bots / spiders that are used to index our content so that it can show up in their respective search engines. These bots follow strict rules, and respond to our requests to slow down if they are hitting us too hard. We love them.
But others are rogues. Pounding our servers without regards to our requests, and scraping our (yours and RateItAll's) content so that they can display it elsewhere and try and make money off of it.
Today, one of those rogue bots was hitting pretty hard - we tracked it back to the University of Singapore. They were hitting us about twice as aggressively as Google, which is pretty aggressive.
So if you guys over there in Singapore are reading this, nice work. You overdid it and now you are permanently banned.
2/7/08
Don't burn yourself....
... on RateItAll's screamingly fast web pages.
RateItAll's new CTO, Mathew, was up most of the night working on our database. He upgraded some software that allowed us to fully use more of the machine's memory, and he fiddled with some settings.
In poking around just now, the site is flying like I haven't seen before.
Nice work Matt.
RateItAll's new CTO, Mathew, was up most of the night working on our database. He upgraded some software that allowed us to fully use more of the machine's memory, and he fiddled with some settings.
In poking around just now, the site is flying like I haven't seen before.
Nice work Matt.
1/25/08
Debrief on Yesterday's Performance Issues
Pretty much all day yesterday there were performance problems with the site related to rating things, and adding new things to the database.
As this site is called Rate It All, these sorts of problems are pretty devastating.
Here's what happened. When unregistered reviewers rate something, we create a "state" that remembers their review so that when they go on to register, their review can post. These folks are stored in the database as "Anons." As you know, no review can go live on RateItAll until there is a registered user behind it. We have to do it this way, or the spammers would overwhelm us.
Well, it turns out that a lot of folks abandon ship once it gets time to join RateItAll.
We hadn't cleaned out the Anons in a little while, and there were 2.7M (yes, MILLION) Anon accounts that were jamming up our database.
This is why things were sluggish yesterday, and we're still working on cleaning these accounts out. Going forward, we'll create a process so that these abandoned Anon accounts are deleted if they do not complete registration...
As this site is called Rate It All, these sorts of problems are pretty devastating.
Here's what happened. When unregistered reviewers rate something, we create a "state" that remembers their review so that when they go on to register, their review can post. These folks are stored in the database as "Anons." As you know, no review can go live on RateItAll until there is a registered user behind it. We have to do it this way, or the spammers would overwhelm us.
Well, it turns out that a lot of folks abandon ship once it gets time to join RateItAll.
We hadn't cleaned out the Anons in a little while, and there were 2.7M (yes, MILLION) Anon accounts that were jamming up our database.
This is why things were sluggish yesterday, and we're still working on cleaning these accounts out. Going forward, we'll create a process so that these abandoned Anon accounts are deleted if they do not complete registration...
1/10/08
Watchmouse: RateItAll Trounces Facebook in Terms of Performance
There was an interesting study recently published by a company called Watchmouse that looked at some of the biggest social networks and how they did in terms of site speed and staying up.
I'm happy to say that RateItAll trounced Facebook in terms of overall performance.
I'm less happy to say that of the 104 major social networks profiled, RateItAll is one of the slowest and one of the worst performing.
When RateItAll is sluggish, it's something that literally makes me sick to my stomach. If you ask RateItAll's developers, or RateItAll's host, or my wife - they'll all tell you.... I turn into a monster when the site is down, or when pages aren't loading as fast as they should, or when things lock up. It's such a basic thing - a slow loading site can ruin even the best features and content.
Over the past year or so, we've tried everything. We've brought in database experts. We bought a massive, state of the art, quad database. We've added load balancing across three web servers. We've removed features and functionality.
I think we're making progress, but we're still not there yet.
But trust me... making RateItAll faster is something that is almost always on my mind and we're going to keep plugging away at it.
I'm happy to say that RateItAll trounced Facebook in terms of overall performance.
I'm less happy to say that of the 104 major social networks profiled, RateItAll is one of the slowest and one of the worst performing.
When RateItAll is sluggish, it's something that literally makes me sick to my stomach. If you ask RateItAll's developers, or RateItAll's host, or my wife - they'll all tell you.... I turn into a monster when the site is down, or when pages aren't loading as fast as they should, or when things lock up. It's such a basic thing - a slow loading site can ruin even the best features and content.
Over the past year or so, we've tried everything. We've brought in database experts. We bought a massive, state of the art, quad database. We've added load balancing across three web servers. We've removed features and functionality.
I think we're making progress, but we're still not there yet.
But trust me... making RateItAll faster is something that is almost always on my mind and we're going to keep plugging away at it.
1/9/08
Kids - Please Learn How to Program
There is nothing more frustrating in my job than not being able to help with hard core technical issues that arise. This morning, we had an issue with a table in our database. It cost us about three hours of uptime.
My role in getting this fixed? Firing off emails and texts to our lead developer and host saying stuff like "it's still timing out." Why? Because I didn't know how to go in and fix it myself.
It was an incredibly frustrating and helpless feeling.
Anyway, my personal motto is "don't be a victim." So, I'm checking out some programming classes at the local community college.
My role in getting this fixed? Firing off emails and texts to our lead developer and host saying stuff like "it's still timing out." Why? Because I didn't know how to go in and fix it myself.
It was an incredibly frustrating and helpless feeling.
Anyway, my personal motto is "don't be a victim." So, I'm checking out some programming classes at the local community college.
12/13/07
Performance might be a little dicey this morning
Trying to battle through getting the site ready for the new home page.
12/6/07
Database Update
.... site will be down for a few minutes.
- OK, step one of two is done. We're still going to have a little more downtime though. Please don't post any reviews until we've finished the last step.
- OK, step one of two is done. We're still going to have a little more downtime though. Please don't post any reviews until we've finished the last step.
11/1/07
Behind the Scenes Stuff
Over the last ten days we've made a couple of important changes to the backend of the site. The result is that RIA pages are loading as fast as I've ever seen them load.
I hired the best database guy I knew to come in and optimize our 1) topic pages; 2) profile pages; 3) item pages; and 4) recent comment pages.
Previously, large user profile pages (no offense GTH) and long topic pages were taking a long time to load. Now they're popping up in under three seconds (and much faster than that on the server side).
Also, we've finally fixed our mail server so that submitting a review doesn't hang while it waits to send out any related alerts.
The site is humming right now - I hope you guys notice :)
I hired the best database guy I knew to come in and optimize our 1) topic pages; 2) profile pages; 3) item pages; and 4) recent comment pages.
Previously, large user profile pages (no offense GTH) and long topic pages were taking a long time to load. Now they're popping up in under three seconds (and much faster than that on the server side).
Also, we've finally fixed our mail server so that submitting a review doesn't hang while it waits to send out any related alerts.
The site is humming right now - I hope you guys notice :)
10/31/07
Running on Half Power
Last night, two of the cooling fans on our quad DB crapped out. The overheated CPUs caused the database server to shut down.
As a short term fix, we disabled the two processors that correlated to the broken fans until we could find some replacement fans.
Currently, RIA's DB is running on only 2 of its 4 processors. This could result in some slowness, especially during peak hours.
The good news is that we've found some replacement fans and will be dropping them into the DB later today, which will allow us to bring all four processors back up. There will be about 20 minutes of scheduled downtime while we make this happen.
Sorry for the inconvenience.
As a short term fix, we disabled the two processors that correlated to the broken fans until we could find some replacement fans.
Currently, RIA's DB is running on only 2 of its 4 processors. This could result in some slowness, especially during peak hours.
The good news is that we've found some replacement fans and will be dropping them into the DB later today, which will allow us to bring all four processors back up. There will be about 20 minutes of scheduled downtime while we make this happen.
Sorry for the inconvenience.
9/4/07
Email Alerts Turned Off
RateItAll's mail server is a piece of garbage. It's always crapping out on us, and worse, it slows down the reviewing process on the site - here's why:
When somebody sets up an email alert on an item or for a reviewer, when a review is posted, the site waits for the email alert to be generated before loading the page. Because the mail server is so crappy, this sometimes takes a few seconds (sometimes more than a few), meaning the page hangs until that connection is made with the mail server.
Well the server has been getting progressively worse. Today, it was taking so long to post a review, that I've decided to just turn email alerts off until we can get a new mail server in place.
So, we're shelling out the $600 for a new server, which should be up sometime tomorrow. Until then, there will be no email alerts (new messages, new ratings, etc.) sent from RateItAll.
The good news is that the reviewing process is super fast now.
When somebody sets up an email alert on an item or for a reviewer, when a review is posted, the site waits for the email alert to be generated before loading the page. Because the mail server is so crappy, this sometimes takes a few seconds (sometimes more than a few), meaning the page hangs until that connection is made with the mail server.
Well the server has been getting progressively worse. Today, it was taking so long to post a review, that I've decided to just turn email alerts off until we can get a new mail server in place.
So, we're shelling out the $600 for a new server, which should be up sometime tomorrow. Until then, there will be no email alerts (new messages, new ratings, etc.) sent from RateItAll.
The good news is that the reviewing process is super fast now.
4/25/07
More Recent Comments Page Problems
It looks like we're having that problem again in which the recent reviews page hangs for logged in users. This is a real pain, as it's the freshest page of the site, and trouble with this page slows down the flow of feedback on our reviews.
We're working on it.
We're working on it.
2/17/07
Fast Enough For You?
The new server is up, and the site is screamingly fast for me right now. Holy smokes - what a difference.
There's an issue with profile pages that we're working on. Hopefully we'll have this fixed shortly.
There's an issue with profile pages that we're working on. Hopefully we'll have this fixed shortly.
Subscribe to:
Posts (Atom)
Posts
