1

We are running a continuous migration (using migrate_plus) from a JSON API which supports last_modified_since as a query parameter. I would like to run this job every hour, and use the time of the previous run as parameter.

What would be the best way to achieve this? I tried tried overriding the variable by adding to the $config array in settings.php, but that is probably not even supposed to work.

1

Yes you can! However, depending on the plugin you are using, it might be picky about what you put in the variable. Here are two examples of variables being used in a migration config:

A file import

source:

  plugin: csv
  track_changes: true
  path: 'public://migration/csv/image_download.csv'
  header_row_count: 1
  keys:
    - image

  fields:
    image: 'Image'
    media_name: 'Media name'

  constants:
    uri_file: 'public://photos'

process:
  uri_file:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/uri_file
        - media_name

  uid:
    plugin: default_value
    default_value: 1

  filename: display_name 

  uri:
    plugin: download
    source:
      - image
      - '@uri_file'

In the above, both constants/uri_file established in the "source" step of the migration, and uri_file passed as @uri_file to the download plugin in "process" step are variables.

The documentation for the migrate_plus process plugin "merge" has another good example:

source:
     plugin: d7_node
   process:
     temp_body:
       plugin: iterator
       source: field_section
       process:
         target_id:
           plugin: migration
           migration: field_collection_field_section_to_paragraph
           source: value
     temp_images:
       plugin: iterator
       source: field_image
       process
         target_id:
           plugin: migration
           migration: image_entities_to_paragraph
           source: fid
     paragraphs_field:
       plugin: merge
       source:
         - @temp_body
         - @temp_images
   destination:
     plugin: 'entity:node'

In both these example, the vars are used in arrays. That's just what the plugin expects, not something inherent to using variables.

  • Thank you for this answer, I had completely overlooked this! In your examples, it looks like the variables are all "internal to" the migration (created in one source, and given to the field as a source). In my case, I would like to have a variable in the url, it is now like this: api.hel.fi/linkedevents/v1/… And I would like to add last_modified_since=@time-of-last-migration, which could be the highWater field. – Bernt Andreas Drange Apr 7 '17 at 8:44
  • Ah, my apologies I misunderstood. In this case, you will probably have to write a small custom source plugin to capture that piece of the url for use in the migration. There might be an existing plugin that could do this, but given the specificity you will need, a custom plugin is probably the way to go. – Caroline Casals Apr 7 '17 at 17:08
  • Thanks for the pointer. I'll try that as soon as I find the time and get back to you. – Bernt Andreas Drange Apr 10 '17 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.