I have a CSV file with approx. 70.000 records that I want to import into Drupal 7. Once this is done the client will provide a new CSV file on a daily base that has to sync the updated rows to Drupal.
If you need an
update feature of your records i'm not sure how you'd get around scanning for Diffs to find changes -- this takes time.
I've used Migrate to upload simple CSV files and tables. Have you looked at Migrate 2.6
Hash() function to track data changes in stuff to perform only updates from your source data?
With Migrate 2.6, we have introduced the 'track_changes' option to MigrateSource. If you pass 'track_changes' => 1 in your options array on your source constructor, Migrate will hash the source data after prepareRow() is called and save the hash in the map table on initial import. On subsequent imports, it will again hash the incoming source data and compare it to the saved hash - if the hashes don't match, the object will be reimported and the new hash saved, otherwise it will be skipped. Of course, this is a slower method than highwater marks, but when you must pick up changed records and don't have a good highwater mark field in the source data, this is the way to go.
Is there any reason you cannot import your CSV file into MySQL tables and then import your data from there using Migrate or a Drush script? Once in tables you could partition the data/tables easily to run the inserts more concurrently for a speedup.
I hit a .NET webservice for 100,000s of records for campus data at an EDU. I found that parsing the data linearly took me on the order of an 1 hour from top of the XML file to the bottom. By requesting each department individually at-the-same-time (get ~60 XML files at once instead of 1 big xml file) and processing in parallel up to 8 threads (via Ultimate Cron) the same process now takes ~5 minutes.
At some point linear processing of data simply fails and you have to find other means.
You could bypass Drupal directly and sync the file directly to the database. This would give you the best possible performance. Here's one way of doing it http://www.tech-recipes.com/rx/2345/import_csv_file_directly_into_mysql/
I am not sure what "forever" means in terms of actual time spent. However, i have used the user import framework and user import framework plus module and it works fine.