Drupalgeddon2 happened! We got all but two of our projects updated within an hour, with those remaining trickier two fully patched another hour later. The key was planning the right process using the right tools. We actually use these tools for regular deployments every day, but speed was essential for this security update. Here's what we did, since some of you may be interested.

1. Our on-call developers split up the various sites/environments/projects that would need updating amongst themselves, using a simple online shared spreadsheet.

2. Ahead of time, we prepared pull requests for sites that simply use Drush make files to specify core versions. (We prefer to keep 3rd-party code, including Drupal core, out of our main project repos, by using composer or Drush make to bring in those dependencies during a build step, executed by Jenkins.) The diff for these would be something as simple as this:

diff --git a/build/stub.make b/build/stub.make
index 03599c39..ff084895 100644
--- a/build/stub.make
+++ b/build/stub.make
@@ -3,7 +3,7 @@ api = 2
 
 ; Drupal core.
 projects[drupal][type] = core
-projects[drupal][version] = 7.57
+projects[drupal][version] = 7.58

3. We monitored all the potential places that the security fix could become available. We noticed it first appear from https://ftp.drupal.org/files/projects/drupal-7.58.tar.gz (we just guessed the filename, as we knew what the release number should be). This meant we were even a few minutes ahead of public announcements actually being made.

4. We assessed the changes that make up the security fix immediately, so we knew whether to fast-track patches directly onto live servers, or if we can use our normal build process. We decided almost all of our sites could be done normally, and we kept an eye out for whether the build steps might become a bottleneck (but we knew that would probably be fine, which was indeed the case). The fix was trivial enough that we didn't need to add any testing to what has already been done by those producing it for Drupal core. In any case, this security risk was so critical, that it was better to risk breaking site functionality than to delay applying the fix. We now know that there were no issues introduced by the changes anyway.

5. We merged any of those pre-prepared pull requests that could be done immediately.

6. Projects with a composer-based workflow (i.e. Drupal 8 ones) would include a hash change in the composer.lock file, so they couldn't be done ahead of time. As soon as the update was available, we ran this command for each project, which relied on correct version constraints already being in place (e.g. "drupal/core": "~8.4.0":

composer update drupal/core symfony/* --with-dependencies

We included all symfony dependencies because we were aware that they can block updates - see Jeff Geerling's excellent write-up. For most sites, that simply changed the composer.lock file to use the new Drupal version, so that could be committed & pushed immediately. Some sites would get a files from core updated, which we wanted to avoid changing (e.g. robots.txt), so we'd revert those.

7. A few of our projects (usually ones adopted from previous owners) keep the whole site codebase (bespoke code + Drupal core & all dependencies) within their main repo. The changes would be very quickly reviewed manually then committed & pushed. For those, the build process is then mostly about verification & deployment.

8. Most of our projects have a build process (using Jenkins) that gets immediately triggered on new commits being pushed to their branches that production/staging environments use. This usually means building a 'full' codebase that includes our bespoke repo, together with the dependencies specified by the composer or drush make files. We use Docker containers and appropriate version constraints to ensure the build process are correct for each project's actual target servers. (For example, building for the appropriate version of PHP, as some dependencies may be different for PHP 7.x or 5.x.) As soon as the full codebase is built, that usually gets automatically deployed to the live sites, and then scripts are ran to apply any necessary updates or clear/rebuild caches. (These scripts, as well as the build tasks themselves, are all part of each project's main code repo.) Builds all run in parallel, so we could update many many sites at the same time.

9. We use Aegir to manage some projects - in particular, one hosts around 1000 sites. Doing the 'correct' thing and building a new platform and migrating each site would have taken too long, so we went ahead and directly applied the patch to the platforms of code from the command line.
   We have since updated the git repo for the project as above, and done a 'correct' migration of the entire platform in Aegir.

10. We would then quickly check the live site is actually running the updated version of Drupal, which is reported under /admin/reports/status .

Using this process, all but two of our sites were updated within an hour. As you can see, this relies on having a strong and reliable build/deploy process set up. Admittedly, there's a few sites which we have to do a little more manually as we don't have the same level of control over their servers (e.g. where we work together with a team at a client that manages that part of a project). Most of those required SSH and a git pull.

We also encountered these extra issues along the way. What issues did you run into? Let us know in the comments, let's all learn from each other in case there's ever another tricky core update!

• A site that we consult for, but aren't the primary developers for, uses some core patches, which would need some re-rolled versions to be compatible with the newer version of core. Those patches were just removed initially to get the security fix out quickly, as that was way more important than the functionality that those patches were fixing.

• We use composer gavel to ensure we are all running the latest version of composer, to reduce merge conflicts in composer.lock . That requires PHP 7, but we had one D8 site on PHP 5.6 (enforced via the platform section of composer.json), so I had to find a workaround, now fixed here.

• We had to be creative to deploy the fix to one site, which we didn't have proper server access to. The client's own team manages its deployments, but they were not available. The next article in this series covers that special case, if you're interested! It wasn't one for the faint-hearted...