Using Migrate API with a multi-page / paginated source

Question

I have a working module pulling a JSON feed into Drupal 8 using the Migrate API. However, the feed is paginated, and I am only getting the first page of results.

For example:

{
    "meta": {
        "count": 58964,
        "next": "http://api.example.com/endpoint/?page=2",
        "previous": null
    },
    "data": [
        {
            "id": "id:1",
            ...
        },
        {
            "id": "id:2",
            ...
        },
...

I need to go through all pages import them all, but I can't see how to tell Migrate to iterate through the pages.

One option is to use a two-step process: first download all the data to a local file and then import that into Drupal. It would be nicer if Drupal 8 Migrate API could fetch and import all the pages itself.

Answer 1

https://www.drupal.org/node/2640516 is the issue to add this support to migrate_plus.

In the meantime, in a case where I knew up front how many pages there were, I simply listed the pages in the urls configuration:

source:
  plugin: url
  data_fetcher_plugin: file
  data_parser_plugin: json
  urls:
    - http://www.example.com/api/search?query=blahblahblah&rowlimit=500
    - http://www.example.com/api/search?query=blahblahblah&rowlimit=500&startrow=501
    - http://www.example.com/api/search?query=blahblahblah&rowlimit=500&startrow=1001
...