I'm using migrate to import 2800 blog posts. Each have a header image which is getting imported. I'm using Rackspace Cloud Files as my CDN so the system is copying the image files up to Rackspace Cloud Files, but after it's been running for awhile I start getting this error:

(/home/drupal/html/sites/all/libraries/rackspace/php-cloudfiles/cloudfiles.php:608) Invalid response: (curl error: 35) error:02001018:system library:fopen:Too many open files: Failed [error] to obtain valid HTTP response.

My -guess- is that the migrate is working too quickly and the file transfers are happening in parallel or something and I'm hitting a limit on the number of open files for Rackspace. I'm not exactly sure.

I'd like to try adding a delay in between each node. There are only 2800 and I can let it run for a little while, so I don't mind the delay as long as it'll let me complete the migrate.

  • Do you have any code to share on how your doing this using Migrate ? What libaries / functions are you using ...
    – tenken
    Dec 6 '12 at 6:37

In your migration scripts you can use either prepare() or complete() (http://drupal.org/node/1132582) and add in sleep(). It's a hacky solution but it should give you enough time between processing each row for Rackspace to catch up.

public function prepare($node, stdClass $row) {
  sleep(30); //wait for 30 seconds
  • I'm experimenting with sleep(1) for now. :) Thanks. Dec 6 '12 at 21:33

I discovered the true problem (although I'm still accepting the other answer because my question was specifically about slowing the migration down).

I believe there is a bug in the rackspace cloudfiles api which causes connections to not be closed properly. In normal, web-based, circumstances this isn't a problem because at the end of the processing all of the connections are closed and you never usually open enough connections to have a problem.

However, I'm running the migration via drush from the command line which means the process isn't ending for a VERY long time after A LOT of connections and eventually I hit the limit for the max number of open "files."

Please see this question for the solution I used to avoid the problem entirely: Migrate with Cloud Files fopen:Too many open files

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.