1

I have a scraper I created that aggregates data found on various sites, feeds, and databases. So that I don't hammer the servers to death that I am connecting to I have it set to sleep 5 seconds between requests. When it finds what I need it saves it as a new node. I have program set to fire on a cron run using hook_cron().

It's works fine but now I have added significantly to the sources I am searching through. The problem is this process could days, especially doing it in series. I have even had the case be that I get an error that cron tried to run while cron was already running.

How can I make this process go more efficiently? More quickly? And not depend on cron? Before I wrote a module for this I simply created several scripts - one script for each source - and I fired them off simultaneously in terminal windows.

How can I do this with my module? I know I could run php's exec() function. In fact this is the commanad was calling with it when I was doing it manually:

php myscripts/search1.php < /dev/null &
php myscripts/search2.php < /dev/null &

When I try this in the module I get bootstrap errors.

When I wrote the module my individual scripts simply became functions in the module. I'm happy to go back to firing off the scripts simultaneously if someone can help me understand how to do that correctly in the module. Or, if you have another suggestion on how to do this quickly and efficiently, I'm all ears. Realistically, I need to process a couple million items of content a week.

Improve this question
3
  • You may need to give more information about your site. You're sort of outside the scope of a regular 'ol website. What does your traffic look like, etc.?
    – vintorg
    Commented Jan 18, 2013 at 21:25
  • That was probably a bit unclear above. It actually does not save the 2 million items, just scans through them, and, if all my criteria are found, then saves that data as a new node. That could happen 1 out of 1000 times or one out of 10,000 times. It's the connecting to the sources and reading them that is the issue. and average of 5 to 8 seconds over 2 million one by one takes a loooooong time!! :)
    – blue928
    Commented Jan 19, 2013 at 0:30
  • I see. For the data you are scanning through, are they on the filesystem, or in another database?
    – vintorg
    Commented Jan 19, 2013 at 2:26

2 Answers 2

Reset to default
2

To fix the bootstrap issue you need to change the CWD (Current Working Directory) to where index.php is located. So in your case you are one directory deep so adding this to the top of your scripts inside of "myscripts" should make them work

// Change the directory to the Drupal root.
chdir('..');

If you were 2 directories deep the code would look like this chdir('../..');. If the errors had to do with php notices/warnings and things of that nature I would look at scripts/drupal.sh (checkout the "define default settings" section) or running your commands/functions with Drush.

There are a couple of options to help with dealing with millions of items of content per week. My recommendation is the HTTP Parallel Request & Threading Library. It will help with downloading things in parallel and doing things in background processes. As an example Link Checker recently integrated with it http://drupal.org/node/380052. With HTTPRL, Link Checker runs _linkchecker_check_links in a new process independent of cron, issue parallel http requests, and runs a callback inside of the event loop allowing work to be done locally while we wait for things to download.

The other options (That I know of) for dealing with LOTS of items are Elysia Cron and Background Process. The stock Batch API will be too slow if your looking to do millions of things per week (there are only 604,800 seconds in a week).

Improve this answer
2
  • Great info, thank you so much! All of these are great directions and I look forward to digging into them. I wonder if Elysia Cron would not be a good start? If I understand the README and API examples on the project page correctly each of my functions now can simply be setup as a separate job that I can then execute simultaneously through the callbacks in the job items array? so if I setup one to fire every minute then ultimately they are all running simultaneously?
    – blue928
    Commented Jan 19, 2013 at 1:26
  • Elysia Cron will let you manage multiple cron tasks. It's the easiest and simplest so I would start there. If you wish to speed up the execution of that task, I would checkout httprl as it will allow you to download in parallel and run node save inside of the event loop with the use of callbacks. When talking about millions of items, doing things in parallel will save a lot of time and server resources.
    – mikeytown2
    Commented Jan 19, 2013 at 1:30
1

Maybe you can take a look to the Batch API instead of cron.

http://api.drupal.org/api/drupal/includes%21form.inc/group/batch/7

Or you can use Drush : http://drupal.org/project/drush With Drush you can organise your cron like you want, in the order you want ...

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.