batch api for large batches

Question

After a form submit, can I create multiple batches based on the maximum table size for the serialized batch string and how can this be done?

I am using a batch process to pass users to an external site using their API.

For each user I set $batch['operations'][] = array('process_funcion', array(LIST OF VARIABLES));

The batch processing works for upto 5000 users; anything over that I get the following error:

Access denied
You are not authorized to access this page.

I thought this had to do with max_packet size, but I am not to sure of that anymore.

The batch is being initiated after a form is being processed.

Do you have any ideas on how I can process 5000 users at a time, or why I am getting this error?

I think that the insert to the batch table is to big, so I am trying to split this into separate batches like this.

$contactCount = 1;
foreach ($contacts as $contact) {
  $batch['operations'][] = array('_contact_add_batch', array($module, $acc_id, $Group_ID, $verb,$contact));
  //Due to max_packet_size restrictions, need to set multiple batches
  if (($contactCount % 2000) == 0){
    batch_set($batch);
  }
  $contactCount++;
}
// For the last set of batches, in the event that there is not 2000, lets add them also.
if (!empty($batch['operations'])) {
  batch_set($batch);
}

I am still getting the Access Denied error.

Answer 1

Batch API in Drupal 6 serializes everything into one row, causing the problem you've seen. You need to do manually what Batch API does for you automatically in Drupal 7: use a queue. The Drupal Queue module is a Drupal 6 backport of the Queue API that is in core for Drupal 7.

Basically, queue the items you want to process, and then only pass the queue ID to Batch API. Batch API then fetches the queue and processes the queue part by part until it's done. See batch_example_batch_2 from the Examples project for reference.

The Examples project also has an example module for queue usage, see: http://drupalcode.org/project/examples.git/tree/refs/heads/7.x-1.x:/queue_example