Feeds importer incremental URL parameter

Question

I am trying to achieve incremental URL changing to avoid creating thousands of importers. For example if the feed URL contains the param ID=1, i need to loop until ID=1000 and import the provided JSON with the same rules. For example..

1. https://domain.com/api/v2/bios.json?bio_id=15&with_faces=true
2. https://domain.com/api/v2/bios.json?bio_id=16&with_faces=true

...and so on...

Is there ANYTHING to allow me do that? Thanks in advance.

Answer 1

You want to take a look at the Web Scraper Example.

https://www.drupal.org/project/example_web_scraper

In Feeds Importer you want to look into the Robots spider setup, which will follow links and patterns, so a single importer can create 10's, 100's or even 1000's of new Nodes.

The example scrapes the Stanford Events website - spidering the page, following the links and creating a new node for each item. You can also run the importer on single event pages.