I have a scraper I created that aggregates data found on various sites, feeds, and databases. So that I don't hammer the servers to death that I am connecting to I have it set to sleep 5 seconds between requests. When it finds what I need it saves it as a new node. I have program set to fire on a cron run using hook_cron().
It's works fine but now I have added significantly to the sources I am searching through. The problem is this process could days, especially doing it in series. I have even had the case be that I get an error that cron tried to run while cron was already running.
How can I make this process go more efficiently? More quickly? And not depend on cron? Before I wrote a module for this I simply created several scripts - one script for each source - and I fired them off simultaneously in terminal windows.
How can I do this with my module? I know I could run php's exec() function. In fact this is the commanad was calling with it when I was doing it manually:
php myscripts/search1.php < /dev/null & php myscripts/search2.php < /dev/null &
When I try this in the module I get bootstrap errors.
When I wrote the module my individual scripts simply became functions in the module. I'm happy to go back to firing off the scripts simultaneously if someone can help me understand how to do that correctly in the module. Or, if you have another suggestion on how to do this quickly and efficiently, I'm all ears. Realistically, I need to process a couple million items of content a week.