Setting up a Drupal API site for module developers

The code that runs http://api.drupal.org is of course Drupal, and it is essentially just scanning the code it's told to and displaying it in a nice format. You can quite easily set up your own API site that you can use to scan your own custom code, or if you're a module developer, your module's documentation (you do have documentation in the code right?)

I'm going to outline how we can use Drupal and Jenkins to build a really nice system for creating an API site that will get updated on-demand, and will manage itself.

The Drupal site

The API module is the key to this, and orchestrates all the cool code scanning and friendly display of your documentation. It is just a standard Drupal module, and has pretty good instructions for getting it set up but you can automate all that with a Drush make file. One is attached at the end of this post, essentially it just assembles the required components, patches the bits it needs to, and then adds some magic. I'm going to talk through the magic in detail.

The magic

I'm going to use Jenkins to keep the code I want to scan up to date, and to force my API to re-scan the code when it does. To allow this I'm going to make two customisations to the API module.

The first is to enable the API module to scan code in a hidden directory, basically Jenkins is going to add the code to a directory that is hidden (at least it was on my mac) and we still want it scanned.

In the parser.inc file around line 1034 there is a line in the api_scan_directories function:

foreach ($files as $path => $file) {
  if (strpos($path, '/.') !== FALSE) {
    continue;
  }

This looks for a directory that has a '.' at the start in any part of the path. I'm just going to comment out the lines so that they look like:

foreach ($files as $path => $file) {
  //if (strpos($path, '/.') !== FALSE) {
  //  continue;
  //}

Great, API module can read hidden directories.

Next, is providing extra Drush integration for the API module. I want a command that will allow me to add new bundles of code to scan from Drush, these are called 'branches' in the API module. The Drush command is pretty simple:

function drush_api_add_branch($project, $branch_name, $branch_title, $directory) {
  // Try to load the branch if possible.
  $branch = api_get_branch_by_name($project, $branch_name);
  if ($branch) {
    if (!empty($branch->data)) {
      $branch->data = unserialize($branch->data);
    }
    else {
      $branch->data = array();
    }
    // This is an update.
    $branch->title = $branch_title;
    $branch->data['directories'] = $directory;
    $branch->data['excluded_directories'] = '';
  }
  else {
    // New branch.
    $branch = new stdClass();
    $branch->branch_id = 0;
    $branch->type = 'files';
    $branch->project = $project;
    $branch->branch_name = $branch_name;
    $branch->title = $branch_title;
    $branch->data['directories'] = $directory;
    $branch->data['excluded_directories'] = '';
    $branch->status = TRUE;
  }
  api_save_branch($branch);
  drush_log('Saved branch.');
}

You can see a patch that adds the command here: http://drupal.org/node/1186230, and don't forget this is all done for you in the Drush make file.

Install Drupal

So, build the code, either with the make file, or manually, and then you'll want to install Drupal as normal. Then enable the API module, you might also want to turn off a lot of other modules, and set the site's homepage to api as well as setting up the permissions for the API module. You will probably want to move the two blocks that API module provides into a region visisble in your theme, navigating without them is going be a challenge otherwise!

Drush alias

We're going to be running drush commands from Jenkins, but we need to be able to specify the API site, the easiest way to do this is with an alias. From the terminal navigate to the site specific directory of your API site and running the following:

drush site-alias your.api.site.fqdn

This will produce something like:

$aliases['your.api.site.fqdn'] = array (
   'root' => '/Users/steven/Sites/api',
   'uri' => 'your.api.site.fqdn',
);

But with the details of your site instead. Save this in file called your.api.site.fqdn.alias.drushrc.php in .drush folder in your home directory. You also need to add an opening PHP tag to this file too.

To make sure that you've got this set up correctly, run the following on the terminal:

cd ~;drush @your.api.site.fqdn status

You should get the standard Drush status output.

Jenkins

The final piece of the puzzle is to get Jenkins set up and configure a couple of jobs to do all the hard work. Getting Jenkins installed is pretty straightforward, follow the instructions on their website. Once you've installed Jenkins, add a new free-style project, call it 'API site - cron' or something similar. All you need to do with this project is add a new build step, of 'Execute shell' type, with the following contents:

/path/to/drush/drush @your.api.site.fqdn cron
/path/to/drush/drush @your.api.site.fqdn php-eval "module_load_include('module', 'job_queue', 'job_queue'); \$j = db_result(db_query('SELECT count(*) FROM {job_queue}')); while (\$j > 0) { job_queue_dequeue(); \$j--; }"

Adjusting the path to drush and the alias for your setup. Execute the project and make sure that the commands are executed succesfully. Basically all we're doing here is running cron, and then making sure that all the job queue items are processed.

Now add another project, again free-style, this time we've got a bit more setting up to do. Add some string parameters:

• 'Project' - From the API module: 'Used as both a unique identifier for the project, and as the URL prefix for documentation for this project (path: api/[project]). You can add a branch to an existing project by entering the existing project name here. You can make this branch be part of a new project by entering a new project name here, and then add additional branches if desired.' For example: 'Hostmaster'.
• 'Branch' - From the API module: Identifier for this branch within this project. Used as the URL suffix for documentation pages for this branch. Also displayed as the version name, in the Versions section at the top of function pages and similar pages (variables, constants, etc.).' For example: '6.x-1.x'
• 'Branch_title' - From the API module: 'Used as the tab title for branch tabs, if you have more than one branch within your project. The title of the first branch within the project is also used as the link text in the Other Projects link sections (these links appear at the bottom of API pages such as the functions list and the home page for each project, if you have more than one project).' For example: '6.x-1.x'

Now add the Drupal project's details to the Source Code Management section, for example I add a git repo, URL: http://git.drupal.org/project/hostmaster.git, branch: 6.x-1.x

In the build triggers section, choose to poll SCM, and choose a schedule. Think about the load that you are going to add to drupal.org when you're choosing a schedule, and if you really need to check every minute!

In the build section, add an 'Execute shell' section and add the command:

/path/to/drush/drush @your.api.site.fqdn api-add-branch "$Project" "$Branch" "$Branch_title" "$WORKSPACE"

Then finally, in the Post build Actions section, choose 'Build other projects' and type the name of the first job we created. I've attached a screenshot of one of my projects, so you can see what the settings look like in Jenkins.

Jenkins should build the project, check out the code, and add it to the API site, and then call the first project which will run cron to scan and add the code to the API site. Cool eh? When the code is updated on drupal.org Jenkins will rebuild the project and run cron on the API site to update the documentation.

Making it useful for module development

Getting quick feedback about documentation changes is essential, so I add a new Jenkins project that is a clone of my project for a module's documentation, but I change the git URL to be a local path, and I increase the polling frequency to once a minute. Then I can make my changes, commit them to git, and Jenkins will cause the API site to be updated in minutes. Once I'm happy with the changes I can push them to Drupal.org

Extra hints for OSX

Brian Gilbert of Realityloop sent me the following in an email, which may be useful for those having issues getting Jekins to run cron or the other tasks of the Drupal site.

Here is what I did to get Jenkins running as me (which isn't the default no matter how you install it using the downloaded OS X installer)

sudo nano /Library/LaunchDaemons/org.jenkins-ci.plist

change

<key>UserName</key>
<string>jenkins</string>

or

<key>UserName</key>
<string>daemon</string>

to

<key>UserName</key>
<string>[username]</string>

and save, then

sudo chown -R `whoami`:staff /Users/Shared/Jenkins

then

sudo launchctl unload -w /Library/LaunchDaemons/org.jenkins-ci.plist

sudo launchctl load -w /Library/LaunchDaemons/org.jenkins-ci.plist