Aegir tasks daemon

Aegir is a very clever hosting system for Drupal that sites and provisions them on various servers and does lots of clever things. One of the clever things that it has had for a while is a task queuing system. You can ask Aegir to lots of different things all in one go, and Aegir will queue them up and run them at its own pace. This provides a really good separation from the front-end Aegir website and the back-end Aegir scripts.

We've been using Aegir in production for just under a year, and one thing that has bugged me is that after adding a task you have to wait a while for the queue to get processed and execute the task. This is because Aegir comes with a queue dispatcher that is invoked by cron, and thus by definition, at most once per minute. So, you could be waiting for up to a minute to log in to the site you want to access as the admin user for example, I can't wait that long! But, rarely do I actually want to be able to execute a task every minute, but when I do queue a task I want it to get processed almost immediately. I want a daemon.

Taking inspiration from the Waiting queue project, I coded up a quick drush command that waits for up to an hour to execute a task from the hosting tasks queue. This is called by a simple bash script, which you then keep alive with something like Supervisor. This has the benefit of checking for new tasks every second, but will do so without having to bootstrap Drupal every second (bootsrapping is about the most intensive thing you can do in Drupal, though probably not Aegir).

The project is available from Drupal.org here:

http://drupal.org/project/hosting_queue_runner

There are fairly detailed installation instructions included in the project, and we've been running it in production for a few days without issues. In fact our idle server load has dropped to a third of the level that it was at before using this script, and we don't have to wait for task execution! Though if you add a lot of tasks to the queue you can easily overload your server, so you'll want to look at the rate limiting options within the front-end module.

I guess I'd like to see this get into Aegir as an 'advanced' option maybe, but I'm quite happy for it to live in Contrib land for now.