5

I have a CSV file with approx. 70.000 records that I want to import into Drupal 7. Once this is done the client will provide a new CSV file on a daily base that has to sync the updated rows to Drupal.

I wrote a import with Feeds and the Migrate module, but in both cases it takes like forever to import the 70.000 rows. Is there any other solution to import the data faster in Drupal 7?

Improve this question
1

If you need an update feature of your records i'm not sure how you'd get around scanning for Diffs to find changes -- this takes time.

I've used Migrate to upload simple CSV files and tables. Have you looked at Migrate 2.6 Hash() function to track data changes in stuff to perform only updates from your source data? http://www.acquia.com/blog/migrate-26-framework-changes

With Migrate 2.6, we have introduced the 'track_changes' option to MigrateSource. If you pass 'track_changes' => 1 in your options array on your source constructor, Migrate will hash the source data after prepareRow() is called and save the hash in the map table on initial import. On subsequent imports, it will again hash the incoming source data and compare it to the saved hash - if the hashes don't match, the object will be reimported and the new hash saved, otherwise it will be skipped. Of course, this is a slower method than highwater marks, but when you must pick up changed records and don't have a good highwater mark field in the source data, this is the way to go.

Is there any reason you cannot import your CSV file into MySQL tables and then import your data from there using Migrate or a Drush script? Once in tables you could partition the data/tables easily to run the inserts more concurrently for a speedup.

EDIT:

I hit a .NET webservice for 100,000s of records for campus data at an EDU. I found that parsing the data linearly took me on the order of an 1 hour from top of the XML file to the bottom. By requesting each department individually at-the-same-time (get ~60 XML files at once instead of 1 big xml file) and processing in parallel up to 8 threads (via Ultimate Cron) the same process now takes ~5 minutes.

At some point linear processing of data simply fails and you have to find other means.

Improve this answer
2
  • The problem is not only the update, the initial insert for the records takes about 10 hours. I already looked into Migrate 2.6 for the update with the hash but that doesn't change anything to the initial import. It's not normal you need 10 hours to import 70 000 records. Anyway THANKS! :) – Jenzzz Jul 3 '13 at 9:49
  • If the uploaded insists big files, you can always split the csv file first using some server side software and let Drupal import then as multiple importers. – AKS Apr 18 '14 at 10:38
0

You could bypass Drupal directly and sync the file directly to the database. This would give you the best possible performance. Here's one way of doing it http://www.tech-recipes.com/rx/2345/import_csv_file_directly_into_mysql/

Improve this answer
3
  • Normally, link only answers aren't considered good ones here? Can you summarize this link? Have you used this technique to import to a Drupal database? – mpdonadio♦ Jul 2 '13 at 20:08
  • Yes but I want to use the drupal framework (views etc..) so tis is not posible (it is in my opinion the fastest approach) :-) – Jenzzz Sep 24 '13 at 12:11
  • For comparibility's sake MySQL import would work best. You can always include your custom tables to Views/Drupal. Awesome case, create a new entity based on that table! – AKS Apr 18 '14 at 10:34
-1

I am not sure what "forever" means in terms of actual time spent. However, i have used the user import framework and user import framework plus module and it works fine.

Improve this answer
2
  • Does this module handle nodes in addition to users? Have you imported 70000 records with it? – mpdonadio♦ Jul 2 '13 at 20:07
  • Like I mentioned before I use the migrate framework to, when you do between 3000/4000 records this is no problem, the problem is when you want to import 70 000 records it runs slow :-) – Jenzzz Sep 24 '13 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.