How to use Drush's concurrent execution capability?

Question

I am using Drupal multi-sites (single codebase, multiple sites/*). Along with this, I have started using Drush aliases to manage them:

$ cat sites/all/drush/aliases.drushrc.php
<?php
$aliases['localdev'] = array(
  'site-list' => array(
    'site1', 
    'site2',
    'site3',
  ),
);
?>

This allows me to easily perform actions on all sites:

$ drush @localdev cc all

>> I've also just discovered that I can just use @sites, and forgo the drushrc file.

Doing this, will execute "cc all" on each of my sites in series (one at a time).

I'd like to take this to the next level and try to run these commands on all sites simulantiously. I've been doing some reading, and am under the impression that Drush does indeed support this. The drush_invoke_process() function takes $backend_options, which can contain (from function documentation):

 *      'invoke-multiple'
 *        If $site_alias_record represents a single site, then 'invoke-multiple'
 *        will cause the _same_ command with the _same_ arguments and options
 *        to be invoked concurrently (e.g. for running concurrent batch processes).
 *      'concurrency'
 *        Limits the number of concurrent processes that will run at the same time.
 *        Defaults to '4'.

What I can't figure out, however, is how do I actually use this from the Drush command line. Is there an option I need to pass to Drush, or do I need to set something in a settings file?

Any info will be much appreciated - my curiosity is piqued!

UPDATE

Based on the answers below, I was able to create a simple test that demonstrates Drush's behaviour, and draw some conclusions:

Drush's default behaviour when executing operations on multiple sites is to use concurrent processes:

$ drush @localdev ev "drupal_set_message(time()); sleep(5);"

Continue?  (y/n): y
site1             >> 1360512943      [status]
site2             >> 1360512943      [status]
site3             >> 1360512943      [status]

This is true even when not using aliases, and is also true when using Drush's built-in @sites alias. These two commands yield identical behaviour as above:

$ drush site1,site2,site3 ev "drupal_set_message(time()); sleep(5);"
$ drush @sites ev "drupal_set_message(time()); sleep(5);"

To change the number of concurrent processes (default is 4), the '--concurrency=N' option can be passed in the drush command. For example, if I want serial execution, I can set the number of concurrent processes to 1:

$ drush @localdev ev "drupal_set_message(time()); sleep(5);" --concurrency=1

Continue?  (y/n): y
site1             >> 1360513387      [status]
site2             >> 1360513393      [status]
site3             >> 1360513399      [status]

Answer 1

This worked for me:

drush @site1,@site2,@site3,@site4 cc all --concurrency=4

I'm not sure how concurrent it actually was; the last message about site1 came immediately after the first message for site2, and all other messages were printed sequentially. I did not measure to what extent each cc operation happened concurrently, or to what extent the system might have just been cpu or i/o bound, but it did seem to be nominally working.

Answer 2

For single instance (without site-list):

<?php
$aliases['localdev'] = array(
  'invoke-multiple' => TRUE,
);
?>

For aliases with site-list array it will run concurently even...

After below comments, let's review code for drush_invoke_process:
// - my comment, /* ... */ - shortening provided code.

<?php
function drush_invoke_process($site_alias_record, $command_name, $commandline_args = array(), $commandline_options = array(), $backend_options = TRUE) {
  if (is_array($site_alias_record) && array_key_exists('site-list', $site_alias_record)) {
    /*  $invocations[] - this array filled with command for each site in site-list. */
  }
  else {
    /* aliases not defined or site-list not found.  So $invocations filled by one item. */
  }
  return drush_backend_invoke_concurrent($invocations, $commandline_options, $backend_options);
}
?>

Next called:

<?php
function drush_backend_invoke_concurrent($invocations, $common_options = array(), $common_backend_options = array(), $default_command = NULL, $default_site = NULL, $context = NULL) {
  /* Here building command line happen for each site (invocation). */
  return _drush_backend_invoke($cmds, $common_backend_options, $context);
}
?>

Next will called:

<?php
function _drush_backend_invoke($cmds, $common_backend_options = array(), $context = NULL) {
  /* Some simulating code and fork code */
  if (array_key_exists('interactive', $common_backend_options) || array_key_exists('fork', $common_backend_options)) {
    /* Direct running (interactive or fork) */
  }
  else {
    // Concurrency set to 4 by default. So --concurency just override it by another value.
    $process_limit = drush_get_option_override($common_backend_options, 'concurrency', 4);

    // Next is main call, that run commands as concurent processes using proc_open and streaming:
    $procs = _drush_backend_proc_open($cmds, $process_limit, $context);

    /* Processing of result running of processes. */

  }
  return empty($ret) ? FALSE : $ret;
}
?>