1

I am executing a script by calling a URL using the wget command. I want to execute this script 3 times a day for 2 hours each time. To do so, I check the difference between the start time and the current time.

My problem is that sometimes the script runs for 2 hours, sometime 1 hour, and sometimes just 3 minutes. I don't know why. I have increased memory_limit and max_execution_time. Still, I haven't been able to find the reason.

What my script is doing exactly:

  1. It fetches a data from other site using cUrl. The data are articles, journals and some other user's data.
  2. After getting the data in XML format, it adds each content.
3
  • Ok so you are saving 1000 nodes in one shot. why not do it in parts ? Put the node save script in cron/batch api/queue and save 50/100/200 nodes(whichever runs in limited time) in one shot and in next cron job start off from where you left. This should be an ideal solution since it can handle 10000 nodes as well in future. Suppose you increase max_execution_time to 3 hours now, what will you do when you are in a situation to save 10k nodes ?
    – GoodSp33d
    Feb 13, 2013 at 6:01
  • 2
    Sounds like a case for the Batch API ( api.drupal.org/api/drupal/includes!form.inc/group/batch/6 ) Feb 13, 2013 at 6:03
  • @2-Stroker I am now saving all content in xml file and saving the nodes after fetching the data from file. I want to run my script for 2 hours. Sometimes it runs perfectly. Feb 13, 2013 at 6:07

1 Answer 1

3

You can break up your work somehow. You shouldn't try to process thousands of things at a time and assume nothing will break or you'll run out of time.

I have recently wrote an in-house Drupal 7 project that allows me to query a webservice for XML data very quickly and process 100,000s of XML entries to an SQL DB pretty fast. A full fetch of all remote SOAP based data is ~100,000 SQL rows after import. This takes ~5 mins to complete. From CURL to DB insert.

The Drupal 7 site performs the following:

  • Allows me to fetch a .NET SOAP response for X departments from my university. X any number of department queries from 1 department to 90 departments per school quarter.
    • I used the Background Process module to fetch the CURL requests to local XML files in a Drupal private filesystem.
    • 90 CURL requests with ~2Mb XML responses complete in ~1 minute. (I could request 1 huge xml file of all departments; I break it up for many threads below).
    • The XML files are added to a Drupal queue. Queue's can lease queue items as jobs for processing. A developer can set a lease time per job for any length of time.
  • I then use hook_cron_queue to define a worker function that parses the XML and inserts the data into a database. This happens to be a non-Drupal database that I connect to via a custom db entry in settings.php.
    • I use the Ultimate Cron and Ultimate Cron Queue Scaler modules to actually perform the processing. I set 9 threads in Ultimate Cron Queue to process queue items, and I have set my lease time to literally 2 hours (7600 seconds). Drupal cron runs/checks for work every minute, the lease time assures no thread tramples over work already in progress.
      • with no optimizations a script I'd written using PHP DomDocument with Xpath queries inserting SQL queries to a DB took minutes per XML file. After adding some SQL batch inserts with db_insert each file takes seconds, but I know from progressively optimizing my scripts a job can have a set lease time. Which is what you want here.

In short. Find a way to break-up your workload and dont be afraid to use watchdog log messages in your module code. A Queue is a nice way to store pieces of the whole task to complete in tiny parts. As you run your worker functions you'll be able to find a lease time that works best for your data size and processing times.

I realized your post is about Drupal 6; it doesn't really matter. The core pieces you want are the D6 queue project and hook_cron(). The Ultimate Cron, Ultimate Cron Queue Scaler, and the Background Process modules have D6 versions.

The project described is a private in-house project, and I cannot easily provide code examples. I've tried to outline how to glue this all together.

1
  • Thnx Giving it a try.. ;) Feb 13, 2013 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.