Monit is a great open source tool for managing and monitoring Unix systems. At Lugo Labs, it's one of the first utilities we install for monitoring our and client servers.

Installation

On our Linux flavour of choice, Ubuntu, installing Monit is as easy as:

sh
sudo apt-get install monit

Capistrano

We use Capistrano to deploy our Ruby on Rails applications to the server. This allows us to have the Monit (and other configurations) inside the application repository, and symlink them in the corresponding folders in the server.

First, we make sure we have the correct settings in the Capistrano's deploy.rb file:

ruby
set :application,   'lugolabs'
set :deploy_user,   '[YOUR DEPLOY USER]'
set :website_url,   'lugolabs.com'
set :full_app_name, "#{fetch(:application)}_#{fetch(:stage)}"
set :config_files,  %w(monit)

set(:symlinks, [
  {
    source: 'monit',
    link:   '/etc/monit/conf.d/{{full_app_name}}.conf'
  }
])

The symlinks variable stores the files to be symlinked in the server. The config files are ERB templates, so that we can inject the Capistrano variables declared above. Let's create a task to parse the templates, copy them to the server then symlink them.

ruby
# lib/capistrano/tasks/setup_config.rb

namespace :deploy do
  task :setup_config do
    on roles(:app) do
      # Copy config files
      config_files = fetch(:config_files)
      config_files.each do |file|
        smart_template file
      end

      # Symlink config files
      symlinks = fetch(:symlinks)
      symlinks.each do |symlink|
        sudo "ln -nfs #{shared_path}/config/#{symlink[:source]} #{sub_strings(symlink[:link])}"
      end
    end
  end
end

The smart_template will do the parsing:

ruby
# lib/capistrano/template.rb

def smart_template(from, to = nil)
  to ||= from
  full_to_path = "#{shared_path}/config/#{to}"
  if (from_erb_path = template_file(from))
    from_erb = StringIO.new(ERB.new(File.read(from_erb_path)).result(binding))
    upload! from_erb, full_to_path
    info "copying: #{from} to: #{full_to_path}"
  else
    error "error #{from} not found"
  end
end

def template_file(name)
  if File.exist?((file = "config/deploy/#{fetch(:full_app_name)}/#{name}.erb")) ||
     File.exist?((file = "config/deploy/shared/#{name}.erb"))
    file
  end
end

Let's make sure the Capistrano tasks are loaded within our Capfile:

ruby
# Capfile

Dir.glob('lib/capistrano/*.rb').each { |r| import r }
Dir.glob('lib/capistrano/**/*.cap').each { |r| import r }

Now we can create a file in the shared folder, config/deploy/shared/monit.erb, where will reside the Monit configuration. Monit allows you to declare programs that can be started and stopped; retries starts automatically, times out, etc.. Let's add some programs needed by our application to the configuration file.

Nginx

We can't have an app without a web server, so we need to have that always running:

ruby
# config/deploy/shared/monit.erb

check process nginx with pidfile /var/run/nginx.pid
  start program = "/etc/init.d/nginx start"
  stop program = "/etc/init.d/nginx stop"
  if children > 250 then restart
  if 5 restarts within 5 cycles then timeout

This assumes that we have an nginx startup script running on /etc/init.d.

PostgreSQL

It's vital for our Rails app to have the database running, so we add that to our Monit configuration:

ruby
# config/deploy/shared/monit.erb

check process postgresql with pidfile /var/run/postgresql/9.5-main.pid
  start program = "/etc/init.d/postgresql start"
  stop program = "/etc/init.d/postgresql stop"
  if failed host localhost port 5432 protocol pgsql then restart
  if 5 restarts within 5 cycles then timeout

Sidekiq

We use Sidekiq for the background jobs and and init script to start it on reboot. So that is the program to declare in Monit:

ruby
# config/deploy/shared/monit.erb

check process <%= fetch(:application) %>_sidekiq_worker
  with pidfile <%= fetch(:sidekiq_pid_path) %>.pid
  start program = "/etc/init.d/sidekiq_<%= fetch(:application) %>_<%= fetch(:rails_env) %> start"
  stop program = "/etc/init.d/sidekiq_<%= fetch(:application) %>_<%= fetch(:rails_env) %> stop"

Redis

Sidekiq requires Redis to store the job queues so we need to install it in the server. Redis comes with its own utilities to start and stop, so we declare those in our Monit configuration:

ruby
# config/deploy/shared/monit.erb

check process redis
  with matching 'redis-server'
  start program = "/bin/systemctl start redis"
  stop program = "/bin/systemctl stop redis"
  if failed host 127.0.0.1 port 6379 then restart
  if 5 restarts within 5 cycles then timeout

Websites

Monit can also ping websites to check that they're always running:

ruby
# config/deploy/shared/monit.erb

check host <%= fetch(:website_url) %> with address <%= fetch(:website_url) %>
  if failed
    icmp type echo count 5 with timeout 15 seconds
  then alert

This tells Monit to check the website URL 5 times with 15 second pauses; if the website is still unreachable, then alert the web master.

Alerts

We use SendGrid to email the alerts to us. We'll add the alert configuration directly in the server:

ruby
# /etc/monit/conf.d/email_settings.conf

set mail-format {
     from: monit@lugolabs.com
  subject: monit alert --  $EVENT $SERVICE
  message: $EVENT Service $SERVICE
                Date:        $DATE
                Action:      $ACTION
                Host:        $HOST
                Description: $DESCRIPTION

           Your faithful employee,
           Monit }
set mailserver smtp.sendgrid.net port 587
   username '[YOUR SENDGRID USERNAME]' password '[YOUR SENDGRID PASSWORD]'
   using TLSV1 with timeout 30 seconds
set alert '[YOUR EMAIL ADDRESS]'

Make sure you use the correct SendGrid username and password and your email address.

Conclusion

After changing the Monit configuration we need to restart Monit, so we add a callback to the deploy.rb

ruby
namespace :deploy do
  after 'deploy:setup_config', 'monit:restart'
end

After deployment we run the

sh
cap production deploy:setup_config

task which restarts Monit. In case of syntax errors, Monit will let us know, so we can fix them.

Happy monitoring!