How to skip empty and missing values from a migration

Question

I'm trying to skip fields that are not present in my source file or that are empty, but I keep getting "Migration failed with source plugin exception: Passed variable is not an array or object"

This is what I've tried in YML:

field_contact:
    -
      plugin: skip_on_empty
      method: process
      source: contact_array
    -
      plugin: sub_process
      source: contact_array
      process:
        target_id:
          plugin: migration_lookup
          migration: import_contact_json
          source: Name

and

field_point_of_contact:
    plugin: sub_process
    source: contact_array
    process:
      target_id:
        -  
          plugin: skip_on_empty
          method: process
          source: name
        -
          plugin: migration_lookup
          migration: registry_import_pocs_json
          source: Name

Both of them have the same output (i.e. the error described above).

Can someone point me in the right direction, please?

Answer 1

I have currently able resolve the issue with this work around.

Combination One is source for paragraph value set 1.

Combination Two is source for paragraph value set 2.

field_section_two is paragraph entity reference multi value field.

Migration Look Up Will return array in this format [ 0 => , 1 => ] this will not considered as empty on skip_on_empty plugin , so passing array_filter plugin to skip the process to avoid exception.

combination_1:
    - 
      plugin: skip_on_empty
      source: image_1
      method: process
    -   
      plugin: migration_lookup
      migration: product_image_inset_one
      no_stub: true
      source:
        - ID
        - image_1
  combination_2:
    - 
      plugin: skip_on_empty
      source: image_2
      method: process
    - 
      plugin: migration_lookup
      migration: product_image_inset_two
      no_stub: true
      source:
        - ID
        - image_2
  field_section_two:
    - 
      plugin: callback
      callable: array_filter
      source:
        - '@combination_1'
        - '@combination_2'
    - 
      plugin: skip_on_empty
      method: process
    -
      plugin: sub_process
      source:
        - '@combination_1'
        - '@combination_2'
      process:
          target_id: '0'
          target_revision_id: '1'

Answer 2

It is difficult to help without seeing the structure of your source file (at least 1 sample record). But, here are some pointers:

• At first glance, the error message says something about a source plugin exception. I believe the error might be stemming from the source plugin configuration.
• The approach you are taking with skip on empty seems to be in the correct direction.
• Depending on the data that the field contains, you might have to handle it differently. I usually add the following to debug exactly what is being received by a particular field process. Also, make sure that the value being sent to the migrate_lookup plugin is an array.

# ...
process:
  field_point_of_contact:
    - plugin: callback
      callable: print_r
      source: name
# ...

• Have you tried removing the skip on empty part? I don't remember correctly, but migrate lookup might be skipping empty values by itself.
• An array with empty strings is not considered empty. If your source provides values like [''], you might have to send the array through an array_filter() to remove empties.

I hope this helps. If it doesn't, you might want to share one row of fake data.

Answer 3

I've promised I'll come back when this is solved, but it took a while :) Thank you everyone for helping out. It turned out the issue was the fact that "migration_dependencies" was set to "required", so any missing data in the source file was firing that exception. After hours of pulling my hair out, the fix was to change migration_dependencies to "optional" And no skip_on_empty is needed in this case.

Answer 4

Setting migration_dependencies to optional resolved this issue for me. I was trying all the skips and custom processes only to realize my very simple error and resolution.