upstart + bluepill + unicorn & delayed_job with rvm and bundler: `exec': No such file or directory

Question

I'm struggling a bit with getting my unicorn and delayed_job processes reboot-proof. I've settled on using bluepill as the overall manager as this can easily be started with upstart in Ubuntu. I've created an RVM wrapper for bluepill and the upstart script works well (starts and stops with ease:

# bluepill - process monitor
#
# simple process monitoring tool

description "simple process monitoring tool"

start on started nginx
stop on stopping nginx

expect daemon
#respawn

exec bootup_bluepill load /home/deployer/apps/nzswarranty/current/config/production.pill

Next up comes the bluepill config file:

Bluepill.application("nzswarranty", :log_file => "/var/log/bluepill.log") do |app|
  app.working_dir = '/home/deployer/apps/nzswarranty/current'
  app.uid = "deployer"
  app.gid = "staff"

  app.process("unicorn") do |process|
    process.start_command = "bundle exec unicorn_rails -c config/unicorn.rb -D"
    process.stop_command  = "kill -s QUIT `cat /tmp/unicorn.nzswarranty.pid`"
    process.restart_command  = "kill -s USR2 `cat /tmp/unicorn.nzswarranty.pid`"
    process.pid_file = '/tmp/unicorn.nzswarranty.pid'
    process.start_grace_time = 15.seconds
    process.stop_grace_time = 15.seconds
  end

  app.process("delayed_job") do |process|
    process.environment = { 'RAILS_ENV' => 'production' }
    process.start_command = 'script/delayed_job start'
    process.stop_command  = 'script/delayed_job stop'
    process.pid_file = '/home/deployer/apps/nzswarranty/shared/pids/delayed_job.pid'
    process.start_grace_time = 15.seconds
    process.stop_grace_time = 15.seconds
  end

end

The server has system-wide RVM installed and bundler managing the gems. I should mention this is a Rails 3.1 app.

Basically when I start bluepill without delayed_job already running I get this when it tries to boot it up:

W, [2012-01-05T13:37:55.185626 #28201]  WARN -- : [nzswarranty:delayed_job] Start command execution returned non-zero exit code:
W, [2012-01-05T13:37:55.185780 #28201]  WARN -- : [nzswarranty:delayed_job] {:stdout=>"", :stderr=>"/usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': no such file to load -- bundler/setup (LoadError)\n\tfrom /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'\n\tfrom /home/deployer/apps/nzswarranty/current/config/boot.rb:6:in `<top (required)>'\n\tfrom <internal:lib/rubygems/custom_require>:29:in `require'\n\tfrom <internal:lib/rubygems/custom_require>:29:in `require'\n\tfrom /home/deployer/apps/nzswarranty/current/config/application.rb:1:in `<top (required)>'\n\tfrom <internal:lib/rubygems/custom_require>:29:in `require'\n\tfrom <internal:lib/rubygems/custom_require>:29:in `require'\n\tfrom /home/deployer/apps/nzswarranty/current/config/environment.rb:2:in `<top (required)>'\n\tfrom <internal:lib/rubygems/custom_require>:29:in `require'\n\tfrom <internal:lib/rubygems/custom_require>:29:in `require'\n\tfrom script/delayed_job:3:in `<main>'\n", :exit_code=>1}
I, [2012-01-05T13:37:55.186003 #28201]  INFO -- : [nzswarranty:delayed_job] Going from down => starting

I've tried using bundle exec for this also and it just says it can't find the bundle executable. My suspicion is that the environment isn't being loaded correctly. Any tips? I have an RVM gemset which is loaded from the .rvmrc file in the root of the project. Should I be switching to this gemset in the bluepill config also?

Answer 1

Ok, so it was pretty easy in the end. I think what was happening is that I installed bluepill in the global rvm gemset as root and also created my wrapper to use the global gemset in the environment (meaning bluepill started ok but it didn't see any of the other gems in my proper gemset (warranty)). I basically removed bluepill from the global gemset and installed it in the warranty gemset. I then created the wrapper again:

rvmsudo rvm wrapper ruby-1.9.2-p290@warranty bootup bluepill

I also encountered another strange error when trying to boot up unicorn but realised I wasn't passing the RAILS_ENV in. Here's my final .pill:

Bluepill.application("nzswarranty", :log_file => "/var/log/bluepill.log") do |app|
    app.working_dir = '/home/deployer/apps/nzswarranty/current'
    app.uid = 'deployer'
    app.gid = 'staff'
    app.environment = { 'RAILS_ENV' => 'production' }

    app.process("unicorn") do |process|
        process.start_command = "bundle exec unicorn_rails -c config/unicorn.rb -D"
        process.stop_command  = "kill -s QUIT `cat /tmp/unicorn.nzswarranty.pid`"
        process.restart_command  = "kill -s USR2 `cat /tmp/unicorn.nzswarranty.pid`"
        process.pid_file = '/tmp/unicorn.nzswarranty.pid'
        process.start_grace_time = 30.seconds
        process.stop_grace_time = 30.seconds
    end

    app.process("delayed_job") do |process|
        process.start_command = 'bundle exec script/delayed_job start'
        process.stop_command  = 'bundle exec script/delayed_job stop'
        process.pid_file = '/home/deployer/apps/nzswarranty/shared/pids/delayed_job.pid'
        process.start_grace_time = 30.seconds
        process.stop_grace_time = 30.seconds
    end
end

It's important to note that we need to use bundle exec before our other commands so that we're loading up the full set of gems from the Gemfile.