How we build

Caibangzi.com
Robin Lu (caibangzi.com)
 Introduction
• Personal Investment Management. Tool +
Community
• Founders: We are engineers!
• Technology:
• Ruby On Rails
• MySQL，Linux，Lighttpd ...
 Topics

• Problems we are facing

• Architecture evolution of caibangzi
• Maintenance
• Q&A
 Peaks!

High Peak
Heavy Computation
Visiting Peak, Computation Peak and Cache
expiration Peak are in the same time. :...(
 Evolution - Start
• Launched at March, 2007
• Hardware: 1 x 1U server
• 1xAMD Opteron 250, 2G Memory, 250G
SATA, RAID 1
• Bandwidth
• Shared
• less than 3Mbps
 Evolution - Start
Apache 2 + mod_proxy_balancer

mongrel x 8
All in
one
memcached caibangzi box

rails 1.1.6

MySQL 5
 First Challenge

• May, 2007
• Extremely time consuming actions blocked
the site.
• Performance tuning
• Optimize Rails cache
 Second Challenge

• August, 2007
• We were running out of bandwidth
• Solution:
• Money
• upgrade to 5Mbps
 Server upgrading

• High growth rate after August, 2007

• Hardware upgrading in Sept. 2007
• still 1 x 1U
• 2 x AMD 2214 2.2GHz
• 4G RAM
 Until Sept. 2007
Apache 2 + mod_proxy_balancer

mongrel x 20
Still in
one
memcached caibangzi box

rails 1.1.6

MySQL 5
Something happened in
Oct. 2007

• Stock Index reached 6000

• Daily pv reached 1 million
• Bigger challenge came
 Several Changes We
Made
• Upgrade bandwidth to 10Mbps
• Separate APP server and DB server
• Load Balancer
• Distributed Computation
 2 Boxes
Web Server

event mongrel x 30

APP
memcached caibangzi

rails

MySQL 5 Tianyuanjia
Load balancer
Run with
2 mongrels
behind
balancer
 Load balancer
• Slow actions block the others
with balancer: Apache
• by request HAproxy
• by traffic Nginx
• round-robin Lighttpd 1.4
Note: Nginx fixed it with
‘fair’ balancer later
 Caibangzi Choice of
Balancer
• Lighttpd 1.5 beta
• proxy-core.balancer = "sqf"
• http://redmine.lighttpd.net/issues/show/
1362
• For Rails application, choice of balancer is
sometimes more important than choice of
web server
 More on Web Server

• disable keep-alive
• serve static files with web server instead of
mongrel
• tune configurations according to your web
server documentations
 Distributed solution
• BackgrounDRB
• based on Distributed Ruby (DRB)
• use ruby thread
• high overhead during context switching
• very slow when concurrent
connections higher than 10
 Distributed solution

• FastBackground
• our own distributed computation
solution
• based on Eventmachine
• simple, but fast and works for us
 Rails upgrading

• Early of 2008, we upgraded to Rails 2.0

• expired several plugins
• refactory to be more RESTful
• cookie session
• multi view => wap.caibangzi.com
 Open API

• API Based on OATH

• integrate caibangzi into your own sites
• Other APPs support several F8 clones
• xiaonei.com, hainei.com, UCHome
 Maintenance

• Munin
• easy to use and extend
• Monit
• Crontab
 With all that....
• we handled:
pv/day 1M
rails req/day 1.8M
peak rails req/sec 160
total req/day 5M
peak req/sec 430

It’s the fact. Not the capacity.

Our CPU usage never goes higher than
30%
Q&A