How to identify item_selector for Migrate API JSON imports

Question

Trying to import articles from an external JSON feed of "members" with a structure like this:

{
   "status":"OK",
   "results":[
      {
         "num_results": 10,
         "offset": 0,
         "articles": [
              {
                 "id": "1",
                 "title": "First article",
...

And I haven't been able to isolate the correct values to use for item_selector to get it to point to the array of "articles":

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls: http://example.com/eample.json
  item_selector: /results/members/

The error I continually get in the console is:

[error] Could not retrieve source count from import_your node: Passed variable is not an array or object

Would be grateful for an explanation of exactly how item_selector is defined or a pointer to documentation for the same.

Answer 1

Think I have a solution, posting here in case it helps others. I tried to find a solution using xpath by using JSON browser like http://jsonpathfinder.com/ which gave me a path like this to find a field:

results[0].members[0].id

The source JSON is coming in as an array but it doesn't appear that the way migrate_plus is handling selectors (splitting on the '/' symbol) is able to handle an xpath array designation, which was flagged in this issue https://www.drupal.org/node/2883803.

To get around this, I thought it would be easier if I could just change the source JSON to fit what migrate_tools can handle so I setup a simple router on my site:

import.my_wrapper:
  path: '/json/wrapper/my.json'
  defaults:
    _controller: '\Drupal\import\Controller\WrapperController::MyWrapper'
  requirements:
    _permission: 'access content'

Then built out the controller to use Guzzle to pull down my source URL normally and then flatten it so that migrate_tools can handle it:

  private function PrepJSONData($data) {
    $data_array = json_decode($data);
    $data_array = $data_array->results[0];
    $data = json_encode($data_array, JSON_UNESCAPED_SLASHES);
    return $data;
  }

I then updated my fetcher in my migrate yaml files to point to the route I had defined on my local site:

  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls: https://example/json/wrapper/my.json

Leaving this question open in case anyone else can propose a more efficient approach.