1

I am trying to import some videos into a 8.0.5 site (yes, I know I should upgrade) I have over 200 videos that neeed to be imported into their own individual node. It seems like a perfect opportunity to get my head around the batch api.

After getting all the videos as objects in a sigle array, I then chunk this into batches of 10 using array_chunk() and then set each chunk as its own opperation.

I would have thought that the batch api would have sent each data array as a signle array to my process function. However, it seems that only the first video from each batch is getting processed.

If I add the &$context variable to my process function and dump it out, it just shows me the next object.

Please, can anyone point out what I have done wrong? :(

public function batchStart() {

ini_set('max_execution_time', 0);
$directory = drupal_get_path('module', 'drp_broadcast_videos_importer') . '/broadcast-videos';
$videoImporter = new VideoImporterController();
$videosArray = $videoImporter->getVideosFromDirectory($directory);
// Chunk up the array.
$videoChunks = array_chunk($videosArray, 5);

foreach ($videoChunks as $chunk) {
  $operations[] = [
    '\Drupal\drp_broadcast_videos_importer\Controller\ImportController::processBatch',
    $chunk,
  ];
}

$batch = [
  'title' => t('Processing Batch'),
  'operations' => $operations,
  'finished' => '\Drupal\drp_broadcast_videos_importer\Controller\ImportController::finishedBatch',
  'init_message' => t('Batch is starting.'),
  'progress_message' => t('Processed @current out of @total.'),
  'error_message' => t('Batch has encountered an error.'),
];

batch_set($batch);

return batch_process('/');
}


static function processBatch($video, $context) {
  try {
    $file = (new VideoImporterController())->createNewFileEntity($video->video, $video->filename);

    $broadcastNode = Node::create(['type' => 'broadcast_video']);
    $broadcastNode->setTitle($video->title);
    $broadcastNode->setOwnerId(\Drupal::currentUser()->id());
    $broadcastNode->setCreatedTime($video->date);
    $broadcastNode->set('field_broadcast_pack_name', $video->pack);
    $broadcastNode->set('field_videos', ['target_id' => $file->id()]);
    $broadcastNode->save();

    $context['message'] = $video->title . ' processed.';
    $context['results'][] = $video->id;

  }
  catch (EntityStorageException $exception) {
//              \Drupal::logger('broadcast_videos_importer')
 //                ->error($this
//                  ->t(
//                    'The broadcast video @title could not be uploaded',
//                    ["@title" => $videoObject->title]
//                  )
//                );
//              drupal_set_message($this->t(
//                'The broadcast video @title could not be uploaded',
//                ["@title" => $videoObject->title]));
    }
  return $broadcastNode->id();
 }
2
$operations[] = [
  '\Drupal\drp_broadcast_videos_importer\Controller\ImportController::processBatch',
  $chunk,
];

In the above, $chunk is the list of parameters that will be passed to the batch process method. So if $chunk is an array of 5 videos, your method signature would need to look like this to match.

static function processBatch($video1, $video2, $video3, $video4, $video5, &$context) {

That's probably not what you're hoping for. To fix, just wrap $chunk in array:

$operations[] = [
  '\Drupal\drp_broadcast_videos_importer\Controller\ImportController::processBatch',
  [$chunk],
];

Now your process method signature looks like this:

static function processBatch($videos, &$context) {

And you just need to loop through the videos in each operation.

  • Thanks @Clive ! Hmmm, ok I've done that, but now I am getting out of memory errors when getting about halfway through the second batch even when lowering the batch about to 2. Isn't the batch api supposed to avoid this? – Jimmyb_1991 Aug 29 '18 at 12:18
  • Yes and no - batching is just a way to split a list of tasks into multiple requests, it's still up to you to manage what can physically be done within your server's memory and time limits in a single processing job. It might be that trying to run 2 of the operations you're trying to run, sequentially, is just too much for your server/php config in a single request. You might need to batch them singularly, which to be fair wouldn't be the worst outcome – Clive Aug 29 '18 at 12:21
  • Yep, you are right, thank you! Although I am still running out of memory, but that is because I am trying to use FFMpeg on a 1.5GB video – Jimmyb_1991 Aug 29 '18 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.