1

I am running Migrate updates with cron every couple of minutes. I don't have many nodes (less than 100) but certain of them are data-heavy. Because of the wonky way my XML is written, I am forced to update all data every couple minutes. My site is now taking a real performance hit. Is this because I am running Migrate with cron? Or is there a way to force Migrate to update via hashes or something? I can't use the highwater field because my XML isn't ordered by update time.

2

Updating all nodes every couple of minutes is bound to kill your performance. It's write-heavy and invalidates caches and there is no way around it in any possible CMS, not only Drupal.

If you cannot decide which nodes needs to be updated on XML read, try to decide on export, and export only nodes that changed since last time. Perform full export-import once a day, deep at night.

Other way around it is to write a custom module that would perofrm following steps:

  1. Download XML file to local variable
  2. Compare checksums of current and previous XML
    • If checksums was equal, exit.
    • If checksums was different, perform import.
  3. If changed update XML checksum.

That way you will only write anything if XML was changed - couple of times a day instead of every other minute.

  • Thing is the XML comes from a third party system and I have no control over structure or the data itself. The data is updated several times a day and so the migrate should take place on update of the XML (ie immediately, regardless of time). Since the XML items aren't organised by date updated, I can't use the highwater mark in Migrate. So I'm at a loss of what to do. I only need to update those nodes that have been updated after the last update. – elsheep Jul 30 '13 at 13:14
  • @elsheep does this XML provide any kind of "updated at" field? – Mołot Jul 30 '13 at 13:30
  • @elsheep updated my answer. – Mołot Jul 30 '13 at 13:34
  • Not for the list as a whole, but the individual items included in the file have "updated by" dates. – elsheep Jul 30 '13 at 13:43
  • @elsheep so just force your import module to omit ones that remain unchanged, or use module that allows you to do so. Or see update to my answer. – Mołot Jul 30 '13 at 13:52
1

As Mołot points out, you are doing a lot of writes, which will be a considerable problem, regardless of how you do it.

force Migrate to update via hashes or something

My experience with Migrate is that the module is highly flexible. Even if this particular functionality isn't there already, I'm fairly certain you could add it yourself.

If you absolutely must do all those writes, consider using MongoDB and its corresponding module. It only implements field storage for the Drupal 7 version though, so you'd need to upgrade core for that to be meaningful for you.

Otherwise, look at improving your I/O speed in other ways, such as an SSD disc.

I'm not 100% sure about the details of how transactions are handled one a low level, so take the below suggestion with a grain of salt.

You could also consider looking at an approach where you do the update in the background on a different physical harddrive, (mysql can be configured to do that within the same db-server, not sure about other dbs), and then when the data is updated, initiate a transaction, switch out the old data with the new, and commit. That way you can consolidate the disc writes on the primary disc to a single large one, which should be significantly faster.

  • For some sources like SQL Migrate can save a hash and compare that to the new data to determine if it should be updated. But I think that is only for SQL sources. – elsheep Jul 30 '13 at 13:42
  • Now I see. Consider re-phrasing the question instead of commenting on my answer. I've updated my answer. – Letharion Jul 30 '13 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.