Modify code to use batch api?

Question

I have a custom module written to import data from a Salesforce API and update organic group info by adding/updating/removing members from groups. I originally wrote it as a drush command for ease of development, and now I've moved it to a set of functions initiated from hook_cron() using a cron queue. An additional requirement is that the import process have the ability to be run from a form submission. The process is long running enough that it times out when running from a simple form submission, so I need to use the batch api. Looking at all the examples, it looks like I need to have one function handle each item so that the batch process can call that function for each individual item. Here's what's working so far:

/**
 * Main function to get data from Salesforce and put it into an array.
 * Called both from a form and drush command.
 */
function naa_committee_get_salesforce_data() {
  $base_url = 'https://api/url';
  // The web service expects the body to be in JSON format, so we create it as an
  // array and convert it to JSON.
  $url_data = array(
    "Name" => "GetCommittees",
    "AuthenticationKey" => "my_secret_key"
  );
  $url_data = json_encode($url_data);
  // Set the Content-Type header.
  $headers = array('Content-Type' => 'application/json');
  // Query the URL and get the returned data.
  $data = drupal_http_request($base_url, array('headers' => $headers, 'method' => "POST", 'data' => $url_data));

  $records = drupal_json_decode($data->data);

  $sf_members = array();
  foreach ($records['Records'] as $record) {
    $user_id = $record['NU__Account__r']['Id'];
    $user_name = $record['NU__Account__r']['Name'];
    $email = $record['NU__Account__r']['PersonEmail'];

    // If array doesn't yet exist for this user, create it.
    if (!$sf_members[$user_id]) {
      $sf_members[$user_id]['user_name'] = $user_name;
      $sf_members[$user_id]['email'] = $email;
    }
    // Add committee to committees element.
    $sf_members[$user_id]['committees'][] = array(
      'committee_name' => $record['NU__Committee__r']['Name'],
      'committee_id' => $record['NU__Committee__r']['Id'],
      'committee_code' =>  $record['NU__Committee__r']['NU__CommitteeShortName__c']
    );
  }

  // Pass data to parsing function.
  if (naa_committee_parse_records($sf_members)) {
    return TRUE;
  }
  else {
    return FALSE;
  }
}

/**
 * Parse returned records
 *
 * @param records
 *   An array of committee member records returned from Salesforce.
 *
 */
function naa_committee_parse_records($sf_members) {
  // Loop through returned records.
  foreach ($sf_members as $user_id => $member) {
    // Do an EntityFieldQuery to see if user already exists.
    $query = new EntityFieldQuery();
    $query->entityCondition('entity_type', 'user')
          ->propertyCondition('name', $user_id);
    $result = $query->execute();

    if ($result) {
      // Since user_id is distinct, we're just taking the first
      // one from $result['node'];
      $uid = current(array_keys($result['user']));
      $account = entity_load('user', array($uid));
      $user = $account[$uid];
      $groups = og_get_entity_groups('user', $user);

      // Need to filter groups to make sure they are committees.
      if ($groups) {
        foreach ($groups['node'] as $key => $group_nid) {
          $group_node = entity_load('node', array($group_nid));
          if ($group_node[$group_nid]->type != 'committee') {
            unset($groups['node'][$key]);
          }
        }

        // Check to see if user should be removed from any memberships.
        naa_committee_check_removed($user, $groups['node'], $member['committees']);
      }

      foreach ($member['committees'] as $committee) {
        if (!$member['email']) {
          $name_array = explode(' ', $member['user_name']);
          $member['email'] = strtolower($name_array[0] . '.' . $name_array[1] . '@naahq.org');
        }
        naa_committee_update_membership($user, $committee, $groups);
      }
    }
    else {
      // Doesn't exist, so create user.
      naa_committee_create_user($user_id, $member);
    }
  }

  return TRUE;
}

So what I have now is one function that loops through all of the records, and for each one determines if it exists, and then hands the record off to the appropriate function. In order to do the batch api, do I need to do something like move the code that loops through the results array to naa_committee_get_salesforce_data() and then pass each individual record to naa_committee_parse_records()? Or can I use it with what I have now?

Thanks.

Answer 1

I assume that you have solved your issue by now but I was just doing some batch API work and found your question. Hopefully, this answer helps.

I added code to create the batch in the first function. The second function will be called by the batch API while processing the batch.

/**
 * Main function to get data from Salesforce and put it into an array.
 * Called both from a form and drush command.
 */
function naa_committee_get_salesforce_data() {
  $base_url = 'https://api/url';
  // The web service expects the body to be in JSON format, so we create it as an
  // array and convert it to JSON.
  $url_data = array(
    "Name" => "GetCommittees",
    "AuthenticationKey" => "my_secret_key"
  );
  $url_data = json_encode($url_data);
  // Set the Content-Type header.
  $headers = array('Content-Type' => 'application/json');
  // Query the URL and get the returned data.
  $data = drupal_http_request($base_url, array('headers' => $headers, 'method' => "POST", 'data' => $url_data));

  $records = drupal_json_decode($data->data);

  $sf_members = array();
  foreach ($records['Records'] as $record) {
    $user_id = $record['NU__Account__r']['Id'];
    $user_name = $record['NU__Account__r']['Name'];
    $email = $record['NU__Account__r']['PersonEmail'];

    // If array doesn't yet exist for this user, create it.
    if (!$sf_members[$user_id]) {
      $sf_members[$user_id]['user_name'] = $user_name;
      $sf_members[$user_id]['email'] = $email;
    }
    // Add committee to committees element.
    $sf_members[$user_id]['committees'][] = array(
      'committee_name' => $record['NU__Committee__r']['Name'],
      'committee_id' => $record['NU__Committee__r']['Id'],
      'committee_code' =>  $record['NU__Committee__r']['NU__CommitteeShortName__c']
    );
  }

  // If there is data to process then setup a batch.
  if (count($sf_members)) {
    $operations = array();
    // Add an operation for each user to be processed.
    foreach ($sf_members as $user_id => $member) {
      $operations[] = array(
        'naa_committee_process_record',
        array(
          $user_id,
          $member,
        )
      );
    }

    $batch = array(
      'operations' => $operations,
      // If desired set the function to be executed when the batch finishes processing.
      'finished' => 'naa_committee_batch_finished',  // remove this line if not desired.
    );

    batch_set($batch);
    return TRUE;
  }
  else {
    return FALSE;
  }
}

/**
 * This function is called by the batch API
 */
function naa_committee_process_record($user_id, $member, &$context) {

  // Use native drupal function user_load_by_name rather than EntityFieldQuery
  // to load user or return false.
  $user = user_load_by_name($user_id);
  if ($user) {
    $groups = og_get_entity_groups('user', $user);

    // Need to filter groups to make sure they are committees.
    if ($groups) {
      foreach ($groups['node'] as $key => $group_nid) {
        $group_node = entity_load('node', array($group_nid));
        if ($group_node[$group_nid]->type != 'committee') {
          unset($groups['node'][$key]);
        }
      }
      // Check to see if user should be removed from any memberships.
      naa_committee_check_removed($user, $groups['node'], $member['committees']);
    }

    foreach ($member['committees'] as $committee) {
      if (!$member['email']) {
        $name_array = explode(' ', $member['user_name']);
        $member['email'] = strtolower($name_array[0] . '.' . $name_array[1] . '@naahq.org');
      }
      naa_committee_update_membership($user, $committee, $groups);
    }

    // Add information to the $context for use by the naa_committee_batch_finished function
    // if desired
    $context['results'][$user->uid] = 'User record updated.';
  }
  else {
    // Doesn't exist, so create user.
    naa_committee_create_user($user_id, $member);

    // Add information to the $context for use by the naa_committee_batch_finished function
    // if desired
    $context['results'][$user->uid] = 'User record created.';
  }
}

Hopefully this helps.

Additional information about passing data to the finished function.