So, you have A Big Idea that will Revolutionize The World. Naturally, people will flock to your site and engage with it. Naturally, you are concerned about whether your infrastructure can handle success. You wish to plan for success, it would be foolish not to, eh? Don’t let enthusiasm (or ignorance) lead you to premature over-optimization. After all, your Big Idea has not yet passed the test of time.
Here I will describe the three early stages commonly encountered when scaling a new website or startup application. Each one builds off the previous ones, starting with the lowest-hanging fruit and working up. Continue reading
This is beautiful! Check out this post on Bret Taylor’s blog:
We use MySQL for storing all of the data in FriendFeed. Our database has grown a lot as our user base has grown. We now store over 250 million entries and a bunch of other data, from comments and “likes” to friend lists.
As our database has grown, we have tried to iteratively deal with the scaling issues that come with rapid growth. We did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as we grew, scaling our existing features to accomodate more traffic turned out to be much less of an issue than adding new features.
After some deliberation, we decided to implement a “schema-less” storage system on top of MySQL rather than use a completely new storage system. This post attempts to describe the high-level details of the system. We are curious how other large sites have tackled these problems, and we thought some of the design work we have done might be useful to other developers.