Best practice for node comparison for more than 2000 nodes

Question

I have around 250 content types for a total of 2000 nodes. What I have to do is finding a clone of each node and display those related clones below the node page in a block (just their titles).

So I'm finding clones on basis of same term reference field value.

For example, for the node with an ID of 6783, there is a field of type term reference (tags). This tag field contains 15 target IDs.

Now I will search all other 199999 nodes; if any node has exactly the same target IDs found in that particular tag field**, I will make all those titles populated below /node/6783.

I need to check on three stages: On first stage, check for all the nodes with the same target IDs; on second stage, check for 90% cloning, meaning looking for the nodes that have 90% of the target IDs; on third stage, check for 80% cloning.

If I will search 2000 nodes, it will exceed the query execution time limit.

What would be the best approach so that it will not affect the performance of website too.

Answer 1

I do not know of a query execution time limit, so I will assume you meant php's max execution time.

If the clone search does not need to be instantly updated, a good approach would be to run the query on cron and cache the results, which means running a single query to fetch the data, process it, and save the results for a quicker fetching at a later time. Cron disables the execution time limit and will run indefinitely in the background.

If the site's cron runs once a day, the clone search results will be up to a day old.

Some resources to give some direction:

• hook_cron - do your processing in this hook
• Cache API - set and get cache data