1

After a form submit, can I create multiple batches based on the maximum table size for the serialized batch string and how can this be done?

I am using a batch process to pass users to an external site using their API.

For each user I set $batch['operations'][] = array('process_funcion', array(LIST OF VARIABLES));

The batch processing works for upto 5000 users; anything over that I get the following error:

Access denied
You are not authorized to access this page.

I thought this had to do with max_packet size, but I am not to sure of that anymore.

The batch is being initiated after a form is being processed.

Do you have any ideas on how I can process 5000 users at a time, or why I am getting this error?

I think that the insert to the batch table is to big, so I am trying to split this into separate batches like this.

$contactCount = 1;
foreach ($contacts as $contact) {
  $batch['operations'][] = array('_contact_add_batch', array($module, $acc_id, $Group_ID, $verb,$contact));
  //Due to max_packet_size restrictions, need to set multiple batches
  if (($contactCount % 2000) == 0){
    batch_set($batch);
  }
  $contactCount++;
}
// For the last set of batches, in the event that there is not 2000, lets add them also.
if (!empty($batch['operations'])) {
  batch_set($batch);
}

I am still getting the Access Denied error.

  • You sure that the error message is an Access Denied one instead of an Internal server error ? In most times, 5000 items shouldn't be the problem i guess. I have used it for 20,000 items.. – AyeshK Jul 29 '12 at 17:55
  • @AyeshK It seems that when the batch is being set, the batch string is being serialized and placed into the batch table for processing. I understand that the amount of data being passed is more than what can be held in the table cell and so the serialized string is not actually written to the table. What I am trying to do is set multiple batches to process around 2000 users per batch, but this does not seem to be working. – Beyerz Jul 30 '12 at 8:52
4

Batch API in Drupal 6 serializes everything into one row, causing the problem you've seen. You need to do manually what Batch API does for you automatically in Drupal 7: use a queue. The Drupal Queue module is a Drupal 6 backport of the Queue API that is in core for Drupal 7.

Basically, queue the items you want to process, and then only pass the queue ID to Batch API. Batch API then fetches the queue and processes the queue part by part until it's done. See batch_example_batch_2 from the Examples project for reference.

The Examples project also has an example module for queue usage, see: http://drupalcode.org/project/examples.git/tree/refs/heads/7.x-1.x:/queue_example

  • That sounds like the perfect solution!! I have been trying to work with queues for some time now and the documentation is really lacking. If you have any good links for queue documentation, please could you share them. In the mean time I will give this a try. – Beyerz Jul 30 '12 at 11:13
  • Edited my answer to provide a link to the queue example. You can also read the docs at api.drupal.org/api/drupal/modules!system!system.queue.inc/group/… – Bojan Zivanovic Jul 30 '12 at 11:37
  • Tried this solution and it worked like a charm!! – Beyerz Aug 1 '12 at 6:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.