1

I am migrating some nodes from an API.

The API only accepts 60 requests per minute.

So it means if 60 items are imported in a minute, all other items will not be imported because the API will return 429 error code in that case.

So on the next cron, the same 60 items will be processed and will be updated. All the other items will never be imported.

    function mymodule_cron() {
      /** @var \Drupal\migrate\Plugin\MigrationInterface $migration */
          $migration = $this->migrationPluginManager->createInstance($migration_id);

          $migration->getIdMap()->prepareUpdate();
          $executable = new MigrateExecutable($migration, new MigrateMessage());
          $executable->import();
    }

So what should I do in this case to import all the items?

Is there any queue functionality with the cron which executes migration? So that I can tell to execute 60 items per queue and execute one queue per minute, something like that?

Update:

$request = $this->httpClient->get($url, [
 'headers' => [
   'Content-Type' => 'application/json',
   'Accept' => 'application/json',
   'Authorization' => 'Bearer ' . $this->token,
 ],
]);
$news_item = json_decode($request->getBody(), TRUE);
$source = $migration->getSourceConfiguration();
$source['data_rows'] = $news_item;
$migration->set('source', $source);

They are 300+ items.

2
  • How big of a response is this API returning? How many items is it? Is it JSON? CSV? How big is the response size (MB)?
    – Joseph
    Commented Feb 9, 2022 at 14:56
  • The response is in JSON. Instead of using the URL, I sent a request to the URL and then use the embedded_data plugin to avoid multiple requests for a single node, because I am also migrating the images and taxonomy from that response. Please check the update section of the question.
    – Ahmad
    Commented Feb 10, 2022 at 5:46

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.