0

I have custom source plugin (imports from MongoDB) and when I run migration for the first time like:

drush mim article --limit=10

it works well: 10 articles are being imported and process stops.

But if I repeat that command then only 1 article is imported.

And if I have like 10 imported articles and run again with limit=20 then 10 new ones are going to be imported.

And if I set limit smaller than current number of imported ones then only 1 new row will be imported.

Should command I specified import always 10 new rows (if there are available, unprocessed)?

Any hint is welcome!

(using drush 11.0.7 and Drupal 9.3.12) (edited)

Update:

So it looks like that code which is counting number of processed rows and checking if limit parameter is reached is not working well, that is, it counts also previously processed rows which should not be the case.

2
  • Looks like my source plugin is responsible for skipping processed row (answered by @mikelutz on slack). So looks like in my case processed rows are not skipped, but counted again. But I didn't override next() method. Any example on how to do this?
    – MilanG
    May 16, 2022 at 14:21
  • next() method (which I didn't override) of SourcePluginBase (which I'm using as a base class) should do the checking if row is processed already or not. But looks like it fails. I had an issue before with condition like that, when previously processed are always been processed again and it was resolved by updating drush.
    – MilanG
    May 17, 2022 at 6:37

1 Answer 1

0

I solved it this way:

Original next() of SourcePluginBase has checking if row processing is needed or not this way:

  if (!$row->getIdMap() || $row->needsUpdate() || $this->aboveHighWater($row) || $this->rowChanged($row)) {
    $this->currentRow = $row->freezeSource();
  }

Now, in my class that inherits that SourcePluginBase I added to it prepareRow override:

  /**
   * {@inheritdoc}
   */
  public function prepareRow(Row $row) {
    if (!$row->getIdMap() || $row->needsUpdate() || $this->aboveHighWater($row)) {
      return parent::prepareRow($row);
    }
    return FALSE;
  }

I noticed that for previously imported rows condition $this->rowChanged($row) is true and for new rows that really needs to be processed $this->rowChanged($row) is true. So I removed first one and kept second one.

I'm aware that this is kind of workaround and that there is probably some incompatibility between drush and drupal, but this is the best solution I could do. If someone has better solution/explanation I will be glad to hear it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.