Ben and the world (/)

June 20, 2012

Rails and Resque: how to easily setup with
(http://blog.benjaminlarralde.com
Redis and God
/post/25723459669/rails-
and-resque-how-to-easily-
(http://blog.benjaminlarralde.com
setup-with-redis-and-god)
/post/25723459669/rails-and-resque-how-to-
9 Comments easily-setup-with-redis-and-god)
(http://blog.benjaminlarralde.com
/post/25723459669/rails- My very first blog post will be dedicated to what has been one of my biggest hurdles
and-resque-how-to-easily- when wanting to setup background tasks with Resque (https://github.com/defunkt
setup-with-redis- /resque) on my white label crowdfunding app (http://www.crowdfundingapp.co).
and-god#disqus_thread)

Setting up the workers is fairly easy, but pushing it to production becomes more
difficult, especially if you want to limit the overhead of managing the different
processes.

Eventually, I spent quite a lot of hours browsing the web looking for fixes to my
particular problems. This post synthesizes my findings by describing the final setup
and highlighting a few tricks.

What I wanted to achieve

For this app, I needed a cron job in order to update projects and process pledges. In
addition, I also wanted a background job that would handle sending emails. All the
workers should be managed in one place and deploying should not require extra steps.

The result is a combination of Resque for the workers, Resque scheduler

(https://github.com/bvandenbos/resque-scheduler) for the cron, Redis for storing
tasks, and God for managing all the processes in production.

Prerequisites
Before you go any further, you should already have Resque workers up and running.
The gem doc (https://github.com/defunkt/resque) is already pretty good, otherwise I
recommend this screencast by Ryan Bates (http://railscasts.com/episodes/271-resque)
and this article from Tanel Suurhans (http://www.perfectline.ee/blog/cron-tasks-
for-your-rails-application-with-resque). As for Redis if you’re new with it, Jim Neath
describes fairly well how to set it up and use it (http://jimneath.org/2011/03
/24/using-redis-with-ruby-on-rails.html).

Please note my configuration at the time of writing this post:

Ubuntu 11.10 x64

Ruby 1.9.2p320 (1.9.2p290 on the server) with RVM
Rails 3.2.3

Step 1: God
God is a process that makes sure that the processes it manages are up and running. If
they’re down for some reason, it restarts them. If you want to kill your processes, just
shut down God, it’ll do it for you. Very valuable when the only UI you have is a
command line.

First, start by installing the gem:

 gem install god

Ben and the world (/) Next, we’ll create a daemon so that it runs automatically on startup. Since I’m using
RVM, I had to create a wrapper (https://rvm.io//integration/god/):

rvm wrapper ruby-1.9.2-p320 boot god

Change ruby-1.9.2-p320 by your version of ruby. I found the correct syntax by

executing which ruby and just extracting the name of the ruby directory.

When this is set, we can create the startup process:

sudo vim /etc/init.d/god

I use vim for editing but you can choose whatever you prefer. In this file, copy paste the
following:

#!/bin/bash ?
#
# god Startup script for god (<a href="http://god.rubyforge.org
#
# chkconfig: - 99 1
# description: God is an easy to configure, easy to extend monitoring \
# framework written in Ruby.
#

CONF_FILE=/opt/god/master.conf
DAEMON=/home/ben/.rvm/bin/boot_god
PIDFILE=/var/run/god.pid
LOGFILE=/var/log/god.log
SCRIPTNAME=/etc/init.d/god

#DEBUG_OPTIONS="--log-level debug"
DEBUG_OPTIONS=""

# Gracefully exit if 'god' gem is not available.

test -x $DAEMON || exit 0

RETVAL=0

god_start() {
start_cmd="$DAEMON -l $LOGFILE -P $PIDFILE $DEBUG_OPTIONS -c $CONF_FIL
echo $start_cmd
$start_cmd || echo -en "god already running"
RETVAL=$?
return $RETVAL
}

god_stop() {
stop_cmd="$DAEMON terminate"
#stop_cmd="kill -QUIT `cat $PIDFILE`"
echo $stop_cmd
$stop_cmd || echo -en "god not running"
}

case "$1" in
start)
god_start
RETVAL=$?
;;
stop)
god_stop
RETVAL=$?
;;
restart)
god_stop
god_start
RETVAL=$?
;;
status)
$DAEMON status
RETVAL=$?
;;
*)
echo "Usage: god {start|stop|restart|status}"
exit 1
;;
esac

exit $RETVAL
 Make sure to replace the directory after DAEMON by whatever your shell answers to
which boot_god
Ben and the world (/)
Then:

sudo touch /var/log/god.log # create an empty log file ?

sudo chmod +x /etc/init.d/god # set execute permissions
sudo update-rc.d -f god defaults # register the startup process

The last step is to create a conf file:

sudo mkdir /opt/god ?

sudo vim /opt/god/master.conf

We’ll look into its contents later.

Step 2: Redis
I said I would’t look at it but we do need to create a conf file:

sudo mkdir /opt/redis ?

sudo vim /opt/redis/redis.conf

Copy paste:

# redis conf ?
daemonize no
pidfile /var/run/redis.pid
logfile stdout

port 6379
bind 127.0.0.1
timeout 300

loglevel notice

## Default configuration options

databases 16

save 900 1
save 300 10
save 60 10000

rdbcompression yes
dbfilename dump.rdb

dir /opt/redis/
appendonly no

What’s important here is that we set daemonize to no so that God can still take control
of it, and also logfile to stdout, so we can set a log file in our God configuration.

Step 3: configuring Redis and Resque for God

We’re going to create one configuration file per task. I’ve put them all in my app in a
directory called god under config/.

You’ll notice that I use ENV variables. That makes it easier to share variables among
the multiple files.

For Redis, config/god/redis.god.rb

Copy/paste:
 rails_root = ENV['RAILS_ROOT'] ?
redis_root = "/usr/bin"

Ben and the world (/) # Redis

%w{6379}.each do |port|
God.watch do |w|
w.name = "redis"
w.interval = 30.seconds
w.start = "#{redis_root}/redis-server /opt/redis/redis.conf"
w.stop = "#{redis_root}/redis-cli -p #{port} shutdown"
w.restart = "#{w.stop} && #{w.start}"
w.start_grace = 10.seconds
w.restart_grace = 10.seconds
w.log = File.join(rails_root, 'log', 'redis.log')

w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end
end
end

For Resque:

rails_env = ENV['RAILS_ENV'] ?
rails_root = ENV['RAILS_ROOT']
rake_root = ENV['RAKE_ROOT']
num_workers = rails_env == 'production' ? 2 : 1

num_workers.times do |num|
God.watch do |w|
w.name = "resque-#{num}"
w.group = 'resque'
w.interval = 30.seconds
w.env = { 'RAILS_ENV' => rails_env, 'QUEUE' => '*'
w.dir = rails_root
w.start = "#{rake_root}/rake resque:work"
w.start_grace = 10.seconds
w.log = File.join(rails_root, 'log', 'resque-worker.log'

# restart if memory gets too high

w.transition(:up, :restart) do |on|
on.condition(:memory_usage) do |c|
c.above = 200.megabytes
c.times = 2
end
end

# determine the state on startup

w.transition(:init, { true => :up, false => :start }) do |on|
on.condition(:process_running) do |c|
c.running = true
end
end

# determine when process has finished starting

w.transition([:start, :restart], :up) do |on|
on.condition(:process_running) do |c|
c.running = true
c.interval = 5.seconds
end

# failsafe
on.condition(:tries) do |c|
c.times = 5
c.transition = :start
c.interval = 5.seconds
end
end

# start if process is not running

w.transition(:up, :start) do |on|
on.condition(:process_running) do |c|
c.running = false
end
end
end
end

For Resque Scheduler:

 rails_env = ENV['RAILS_ENV'] ?
rails_root = ENV['RAILS_ROOT']
rake_root = ENV['RAKE_ROOT']
Ben and the world (/)
God.watch do |w|
w.name = "resque-scheduler"
w.group = 'resque'
w.interval = 30.seconds
w.dir = rails_root
w.env = { 'RAILS_ENV' => rails_env }
w.start = "#{rake_root}/rake resque:scheduler"
w.start_grace = 10.seconds
w.log = File.join(rails_root, 'log', 'resque-scheduler.log'

w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end
end

Step 4: God’s master file

We created a file before called master.conf under /opt/god/. It’s time to use it.

ENV['RAILS_ENV'] = "production" ?
rails_root = ENV['RAILS_ROOT'] = "/path/to/your/app/current"
ENV['RAKE_ROOT'] = "/path/to/your/rake/bin"

load "#{rails_root}/config/god/redis.god.rb"
load "#{rails_root}/config/god/resque.god.rb"
load "#{rails_root}/config/god/resque_scheduler.god.rb"

And that should do it! Now, in the terminal, type: sudo service god start This
should start God and all its dependencies. If something seems wrong, check the log file
at /var/log/god.log and fix whatever is needed.

Bonus: Capistrano
The nice thing now is to have your workers restart every time you deploy, so that they
run the latest code. To do so, add the following in your resque.god.rb:

w.transition(:up, :restart) do |on| ?

# restart if server is restarted
on.condition(:file_touched) do |c|
c.interval = 5.seconds
c.restart_file = File.join(rails_root, 'tmp', 'restart.txt')
end
end

This will work assuming you have a task deploy:restart which “touches”
/tmp/restart.txt, which is very likely.

You’ll also need to add this file (https://github.com/cyrilpic/god/blob

/0862ec8d889470b8de09e6b75e50de2c306951d0/lib/god/conditions
/file_touched.rb) to your God config. Save it wherever you want, for instance /opt/god,
and link to it by adding load path/to/file in master.conf.

Last words
This is quite a complex setup and you should expect to run into a couple of issues
depending on your own configuration. Hopefully, this step by step will have made it
less painful. Let me know if that was any useful!

Before we go, I’d like to thanks all the people who I borrowed code from, even though
I’m unfortunately unable to pin point who those were.

Share
Ben and the world (/)
9 Comments Ben and the world !
1 Login

♥ Recommend ⤤ Share Sort by Best

Join the discussion…

Mikhail Klishevich • 10 months ago

Thanks for the post. I have a problem with ENV['RAKE_ROOT']
variable in master.conf. When I set ENV['RAKE_ROOT'] =
'/home/mike/.rvm/gems/ruby-1.9.3-p545@myaksts/bin', I get
the error 'Could not find mono_logger-1.1.0 in any of the sources'.
What value should I set?
• Reply • Share ›

blarralde Mod > Mikhail Klishevich • 10 months ago

With that amount of details I can't really help you much.

all I can say is that it looks like it can't install the
mono_logger gem v.1.1.0, so it might be that you made a
typo in the gem name, that version doesn't exist, or you
need to specify a specific source/git repo for it.
• Reply • Share ›

Mikhail Klishevich > blarralde • 10 months ago

Problem solved :). After I set RAKE_ROOT =
/home/mike/.rvm/wrappers/ruby-1.9.3-
p545@myaksts , everything works! Thanks again
for great manual!
• Reply • Share ›

Sila • 2 years ago

Thank you for this really helpful manual! I had to adjust a few
custom paths but then it works pefect!
• Reply • Share ›

blarralde Mod > Sila • 2 years ago

Glad it helped!
• Reply • Share ›

Алексей Скобликов • 2 years ago

Thank you for detailed manual. I did the rvm-god-resque as you
described. And it works.

However I have a problem, maybe you may have an idea.

It is described at http://stackoverflow.com/quest...

Please pay some attention at my question, it is short, but

intriguing. The resume is that resque being run from init.d god
service is not working properly, when resque job is
communicating with local application through IO.popen. Weird.
• Reply • Share ›

blarralde Mod > Алексей Скобликов • 2 years ago

Hi Aleksei, unfortunately I switched to Heroku so I don't

have to deal with God anymore, and I've never worked
with IO so I'm not really sure what's going on...
• Reply • Share ›

Zak El Fassi > blarralde • a year ago

I'm intrigued to know if you're still at Heroku.
I was there for a while -loved it, then stuff started
growing, and the bills too ... eventually I had to
Ben and the world (/)

< Previous post Next post >

(http://blog.benjaminlarralde.com

/post/27255872468)
Theme by Pixel Union

(http://www.pixelunion.net)