I have script that queries a system called Symplectic for publications of over 30 users (each authored over 300 publications). Most of the publications are authored jointly by more than one user. I have done the following using a combination of Entity API and other methods:
- Programmatically creating publication nodes (with fields i.e. integer, text, lists, taxonomy terms, entity reference, etc).
I would like to avoid the following scenario:
- Creating duplicate nodes of type publication
Every time I create a publication, I would like to check whether the publication already exist or not (most probably yes because, the users have authored so many publications together). I can achieve this when I am creating all publications at the same time :
- I will lookup a structure the stores publication specific properties (i.e. Symplectic object id, doi, publication source id, last-modified-timestamp where source being pubmed, scopus, etc.). If not exist then create publications and update the structure, else reference the user to the already existing publication.
The actual use I am Concerned about is, Updating thousands of publications at a later date after they are created. I have the following options, which one do you think is most efficient:
Use EntityFieldQuery to query for all nodes of type publications (over 2500 nodes) at once, and store the properties useful for comparison in a structure. Use the structure to decide whether publication exist in Drupal, if no then create it, if yes then check for last-modified-date.
Use EntityFieldQuery with condition fields to check if a publication already exist in the Drupal system, if yes then check if update is avaiable, if no then create it. This method would require, N number of quries (2500 publication nodes would result in 2500 calls).
During initial create operation (when no publication node exists), create a structure and store the few publication fields that are expected to be unique (not timestamp) across all publications (i.e. symplectic object id, doi, publication source id, last-modified-timestamp), serialize the structure and store it in the file system. During update phase, load the structure from file system, unserialize it and use it to check whether a publication node already exist in the drupal system, if no then create it and update this structure, if yes then check whether update is required.
My worries in regards to above available options:
Loading over 3000 publication nodes (of course, EntityFieldQuery would retrun all nids, then I will have to use either node_load_multiple or node_load) - Is this efficient? Would I not face memory problems?
2500+ EntityFieldQuery does not seem to be the right choice due to query time. (Attaching conditions might increase the query time i.e. if symplectic id field of publication equals X)
This is my idea, I don't know if it makes sense.
I have Drupal 7 on CentOS 6.5, and a publication content type has the following number of fields: boolean - 2 text - 20 Double field - 3 Taxonomy - 1 List 2, title, body, etc.
What do you recommend based on your experience? I did have a look at Biblio module, it is not the way to go forward for our requirements. I also know that $conditions parameter of node_load_multiple is deprecated in D7.
Sorry for long description, I tried to be as descriptive as possible.