Performance issues with Migrate Cron update

Question

I am running Migrate updates with cron every couple of minutes. I don't have many nodes (less than 100) but certain of them are data-heavy. Because of the wonky way my XML is written, I am forced to update all data every couple minutes. My site is now taking a real performance hit. Is this because I am running Migrate with cron? Or is there a way to force Migrate to update via hashes or something? I can't use the highwater field because my XML isn't ordered by update time.

Answer 1

Updating all nodes every couple of minutes is bound to kill your performance. It's write-heavy and invalidates caches and there is no way around it in any possible CMS, not only Drupal.

If you cannot decide which nodes needs to be updated on XML read, try to decide on export, and export only nodes that changed since last time. Perform full export-import once a day, deep at night.

Other way around it is to write a custom module that would perofrm following steps:

1. Download XML file to local variable
2. Compare checksums of current and previous XML
   • If checksums was equal, exit.
   • If checksums was different, perform import.
3. If changed update XML checksum.

That way you will only write anything if XML was changed - couple of times a day instead of every other minute.

Answer 2

As Mołot points out, you are doing a lot of writes, which will be a considerable problem, regardless of how you do it.

force Migrate to update via hashes or something

My experience with Migrate is that the module is highly flexible. Even if this particular functionality isn't there already, I'm fairly certain you could add it yourself.

If you absolutely must do all those writes, consider using MongoDB and its corresponding module. It only implements field storage for the Drupal 7 version though, so you'd need to upgrade core for that to be meaningful for you.

Otherwise, look at improving your I/O speed in other ways, such as an SSD disc.

I'm not 100% sure about the details of how transactions are handled one a low level, so take the below suggestion with a grain of salt.

You could also consider looking at an approach where you do the update in the background on a different physical harddrive, (mysql can be configured to do that within the same db-server, not sure about other dbs), and then when the data is updated, initiate a transaction, switch out the old data with the new, and commit. That way you can consolidate the disc writes on the primary disc to a single large one, which should be significantly faster.