Recursive Drush make, working with different git branches

Question

Setup

For every site we make a custom Drupal install profile is created, copies of which are built using drush make (through Aegir, though that should not make a difference).

There is a top-level build makefile for downloading Drupal core and our install profile, and the install profile contains a makefile specifying Features to download, each of which have a makefile specifying their dependencies (either custom modules in private git repos or contrib modules).

The build makefile (build-SITENAME.make) specifies the install profile like this:

projects[SITENAME_profile][type] = profile
projects[SITENAME_profile][download][type] = git
projects[SITENAME_profile][download][url] = git.hostname:drupal-sites/SITENAME_profile

Then the install profile's makefile will specify a bunch of Features to download, for example:

projects[blog][type] = module
projects[blog][subdir] = features
projects[blog][download][type] = git
projects[blog][download][url] = git.hostname:drupal-features/blog

Which will then download custom and contrib modules and so on.

The above setup has worked well for a couple of years.

Problem

The problem occurs when trying to use git branches for managing different stages of development, e.g. development, staging and production; and having CI automatically manage Drupal sites for each branch. If a build is triggered for the development branch, drush make has to get the correct branch for every one of our repos it downloads.

Given our multi-repo and recursive makefile structure, how would drush make be used to recursively download the correct branch from our git repos?

Things already tried

1. drush make allows downloading from a git branch, which seemed good for our use-case. For example, for each makefile (in the build, install profile and Features repositories) a line is added that looks like:

   projects[blog][download][branch] = development
   

   When the branch of the repo being downloaded matches the branch of the repo the makefile is in, custom modules specified in the development branch of the 'blog' Feature will be downloaded from that module's development branch. Great!

   However, this causes problems merging the branches of any repo containing a makefile (given our setup that's quite a lot), we will have to be extremely careful not to accidentally merge [branch] = development into master on any of them.

2. Checked drush make documentation for a command-line option to specify the default branch for git repository downloads. Found nothing.
3. Searched the Internet for people with similar quandaries, it seems many are dumping all the dependencies for whole projects into single make files, but that breaks the modularity of Features.

Note: I'm not polling or asking for discussion. I believe there's something silly I'm missing and/or something everyone else is doing that I'm not.

Answer 1

This is a great question! I maintain both Aegir and Drush Make, but I'm not certain that I have a really good answer for you.

1. I think that you're suggestion in (1) is probably the best bet at the moment. It has the virtue of already working, as you've mentioned. The issue with accidentally merging these [branch] attributes is still valid though. The solution though, lies not in Drush Make, but in Git!

   Check out this article. It describes using Git attributes, along with a custom (null) Git merge driver. To summarize, each developer would have to run the following:

   git config --global merge.ours.driver true
   

   This basically means that when an item is flagged to use the 'ours' merge driver, it'll just call /bin/true, hence my earlier reference to it being a null operation.

   But, how do you flag your makefiles to use this driver? That's where Git attributes come into play. Create a .gitattributes file at the root of your repo (i.e., along side .gitignore). Then add the following line:

   build-SITENAME.make merge=ours
   

   You'll probably want to do the same for your feature repos as well.

2. As for solutions leveraging Drush Make, I don't know that there's a solid solution.

   A few years back, I experimented with using overrides trying to do something similar, as described in the Drush Make documentation. The basic idea is that you can change (override) attributes from included makefiles.

   However, in the case of recursive makefiles, they are built separately, instead of being merged into the parent makefile prior to build. As a result, there's no opportunity to override their attributes. This may have changed in the time since I tested it, but I think the underlying mechanics remain largely the same, and so it's unlikely to work.

   Setting a default in a parent makefile would be a nice way to accomplish something like this too. I haven't tested this, but again, I'm pretty sure the default wouldn't apply to the recursively built makefile.

   Instead, an option would be to build your parent makefile without recursion, but rather including the feature makefiles directly. You can now include makefiles directly from within a Git repo (using the new YAML syntax):

   includes:
     # A file from a git repository.
     makefile: "example_dir/example.make"
     download:
       type: "git"
       url: "[email protected]:organisation/repository.git"
       branch: "master"
   

   You'd either want to call drush make with the --no-recursion option, or specify it in your makefiles:

   --no-recursion                   Do not recurse into the makefiles of any
                                    downloaded projects; you can also set 
                                    [do_recursion] = 0 on a
                                    per-project basis in the makefile.
   

I'm sure there are other options available, but those are the ones that I think would least impact your existing workflows. I'd be very curious to see other suggestions.

There is a feature request to allow the use of variables in .make files, which might address such a use-case more directly, if it were implemented. Perhaps we could look for such variables in options passed on the command line, or in environment variables.

That said, in the long run, I believe that Drush Make will be supplanted entirely by Composer, so it might be more fruitful to explore in that direction, if the suggestions above don't work out.

Answer 2

Consider using composer which could be a bit more flexible regarding the recursive dependencies as it can analyze the whole dependency tree which you can set-up and many more.

See example composer.json:

...
    "require": {
        "composer/installers": "^1.0.20",
        "drupal/core": "8.0.*",
        "drush/drush": "8.*",
        "drupal/console": "~0.8",

        "drupal/devel": "8.1.*@dev",
        "drupal/token": "8.1.*@dev"
    },
...

See:

• Composer vs. Drush Make: Which Should You Use?
• Composer template for Drupal projects.

Answer 3

If you're interested in managing different branches, please have a look how this is achieved in ADS distribution (GitHub).

Basically you can set-up CI to load the file appropriate for specific branch, e.g.:

build-SITENAME-BRANCH.make

then in this file you can have specific components of specific branch (e.g. profile) and everything else (e.g. common contrib modules) can be included from different .make files (includes[] = contrib.make). See: build-ads-latest-dev.make.

---

If this won't help, alternatively use editing in-place before running the build which will set the appropriate branches.

For example having this line where you specify your branch:

projects[foo][download][branch] = master

you can alter it via:

ex -s +"%s/master/develop/" -cwq build-SITENAME.make