2

I have a working module pulling a JSON feed into Drupal 8 using the Migrate API. However, the feed is paginated, and I am only getting the first page of results.

For example:

{
    "meta": {
        "count": 58964,
        "next": "http://api.example.com/endpoint/?page=2",
        "previous": null
    },
    "data": [
        {
            "id": "id:1",
            ...
        },
        {
            "id": "id:2",
            ...
        },
...

I need to go through all pages import them all, but I can't see how to tell Migrate to iterate through the pages.

One option is to use a two-step process: first download all the data to a local file and then import that into Drupal. It would be nicer if Drupal 8 Migrate API could fetch and import all the pages itself.

3

https://www.drupal.org/node/2640516 is the issue to add this support to migrate_plus.

In the meantime, in a case where I knew up front how many pages there were, I simply listed the pages in the urls configuration:

source:
  plugin: url
  data_fetcher_plugin: file
  data_parser_plugin: json
  urls:
    - http://www.example.com/api/search?query=blahblahblah&rowlimit=500
    - http://www.example.com/api/search?query=blahblahblah&rowlimit=500&startrow=501
    - http://www.example.com/api/search?query=blahblahblah&rowlimit=500&startrow=1001
...
  • Nice workaround! I don't know exactly how many pages there are, but it works if I put a few more extra than expected. It gives an error with a 404 on the first one not found, and doesn't process any further URLs, which is just fine. – Hugo Feb 22 '17 at 9:38
  • 1
    There is a patch on the issue that @mike-ryan mentions now which seems to work well for us, so even though it isn't committed yet it could be the definitive answer for this question. If only somebody had time to write some tests.. – Bernt Andreas Drange Sep 12 '17 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.