Monitoring Varnish

We put almost all of our Drupal sites behind the excellent Varnish HTTP accelerator, and it gives us a massive performance boost for most site visitors. However it seems to have a tendency to crash without warning and occasionally just dies, leaving our sites down.

We workaround this issue by using another piece of useful kit, called Monit, that keeps an eye on processes on your server and restarts them if necessary.

Installing and configuring Monit is really simple (these instructions are for a Debian based server):

Install

Installation is simple:

apt-get install monit

Configuration

Configuration is simple too, you will have been told by the installer that you should set up the configuration file and then edit a file to enable the Monit monitoring daemon.

Edit the Monit configuration file: /etc/monit/monitrc and edit the section near the top about starting Monit to look something like:

set daemon  60           # check services at 1-minute intervals
    with start delay 30  # optional: delay the first check by 30 seconds

Then add the following to the bottom of that file:

# Check varnish on port 80
check process varnish with pidfile /var/run/varnishd.pid
start program = "/etc/init.d/varnish start"
stop program = "/etc/init.d/varnish stop"
if failed host 127.0.0.1 port 80 protocol http
  and request "/monit-check-url-qwertyuiop"
  then restart
# Uncomment the following line to cause Monit to stop trying
# to bring Varnish back up if it keeps going down.
# if 3 restarts within 5 cycles then timeout

Tweak the location of the pidfile on the second line according to your configuration, the location should be set in /etc/init.d/varnish so check there if you don't know it. Also change the URL requested from monit-check-url-qwertyuiop to something that you will never need to request from your website.

With these two small bits of configuration we're basically getting Monit to start, wait 30 seconds and then every 60 seconds make a HTTP request to 127.0.0.1:80/monit-check-url-qwertyuiop. If the request fails then Monit will restart Varnish.

This should deal with most cases of Varnish dying and result in at most 1 minute of downtime.

Configuring Varnish

Most tutorials will not tell you to do anything special with Varnish, and to just request any old URL with the Monit check, but that would actually be testing the full stack of your application, not just Varnish. We can add a small bit of configuration to Varnish to provide a URL for Monit to check that will be handled entirely in Varnish.

At the top of your Varnish VCL file, usually /etc/varnish/default.vcl add this:

# Include our vcl file for monit.
include "/etc/varnish/monit.vcl";

Then create a new file at: /etc/varnish/monit.vcl and add the following:

# We provide a 'special' URL for monit to hit that will be entirely handled by
# Varnish and never touch a backend. This means that Monit will actually be able
# to test to see if Varnish is up, not if Varnish and Apache are up.

# Define the vcl_recv subroutine, ours will get handled first, then any others.
sub vcl_recv {
  # Change this URL to something that will NEVER be a real URL for the hosted
  # site, it will be effectively inaccessible.
  if (req.url == "/monit-check-url-qwertyuiop") {
    error 200 "Varnish up";
  }
}

Note that here we provide the same URL that we added in the Monit check earlier, these must match, for, hopefully, obvious reasons.

Restart Varnish by running:

/etc/init.d/varnish restart

And you should be able to access the URL you just 'created' in Varnish, using your normal web browser and it should return an error page (though the HTTP error code will be a 200, which isn't an error).

Starting Monit

As the Monit installer should have told you, you should now edit the /etc/default/monit file and follow the instructions in there. Finally start everything up by running:

/etc/init.d/monit start

Testing

The easiest way to test is to run this:

/etc/init.d/varnish stop

And wait a little while to see if Monit brings Varnish back up. If things don't work, then have a read through /etc/monit/monitrc for configuration options for logging and even a web interface for insight into what Monit is doing. Otherwise, pat yourself on the back.

More services

Monit is very versatile and can monitor many services, for example, you may wish to monitor Apache, MySQL or some other service your Drupal sites rely on.

Updated: Varnish 4

If you have Varnish 4, then the code for the Varnish vcl will need to look like this:

sub vcl_recv {
  # Change this URL to something that will NEVER be a real URL for the hosted
  # site, it will be effectively inaccessible.

  if (req.url == "/monit-check-url-qwertyuiop") {
   return(synth(200, "OK"));
  }
}