How to execute a script that takes a long time?

Question

I am executing a script by calling a URL using the wget command. I want to execute this script 3 times a day for 2 hours each time. To do so, I check the difference between the start time and the current time.

My problem is that sometimes the script runs for 2 hours, sometime 1 hour, and sometimes just 3 minutes. I don't know why. I have increased memory_limit and max_execution_time. Still, I haven't been able to find the reason.

What my script is doing exactly:

1. It fetches a data from other site using cUrl. The data are articles, journals and some other user's data.
2. After getting the data in XML format, it adds each content.

Answer 1

You can break up your work somehow. You shouldn't try to process thousands of things at a time and assume nothing will break or you'll run out of time.

I have recently wrote an in-house Drupal 7 project that allows me to query a webservice for XML data very quickly and process 100,000s of XML entries to an SQL DB pretty fast. A full fetch of all remote SOAP based data is ~100,000 SQL rows after import. This takes ~5 mins to complete. From CURL to DB insert.

The Drupal 7 site performs the following:

• Allows me to fetch a .NET SOAP response for X departments from my university. X any number of department queries from 1 department to 90 departments per school quarter.
  • I used the Background Process module to fetch the CURL requests to local XML files in a Drupal private filesystem.
  • 90 CURL requests with ~2Mb XML responses complete in ~1 minute. (I could request 1 huge xml file of all departments; I break it up for many threads below).
  • The XML files are added to a Drupal queue. Queue's can lease queue items as jobs for processing. A developer can set a lease time per job for any length of time.
• I then use hook_cron_queue to define a worker function that parses the XML and inserts the data into a database. This happens to be a non-Drupal database that I connect to via a custom db entry in settings.php.
  • I use the Ultimate Cron and Ultimate Cron Queue Scaler modules to actually perform the processing. I set 9 threads in Ultimate Cron Queue to process queue items, and I have set my lease time to literally 2 hours (7600 seconds). Drupal cron runs/checks for work every minute, the lease time assures no thread tramples over work already in progress.
    • with no optimizations a script I'd written using PHP DomDocument with Xpath queries inserting SQL queries to a DB took minutes per XML file. After adding some SQL batch inserts with db_insert each file takes seconds, but I know from progressively optimizing my scripts a job can have a set lease time. Which is what you want here.

In short. Find a way to break-up your workload and dont be afraid to use watchdog log messages in your module code. A Queue is a nice way to store pieces of the whole task to complete in tiny parts. As you run your worker functions you'll be able to find a lease time that works best for your data size and processing times.

I realized your post is about Drupal 6; it doesn't really matter. The core pieces you want are the D6 queue project and hook_cron(). The Ultimate Cron, Ultimate Cron Queue Scaler, and the Background Process modules have D6 versions.

The project described is a private in-house project, and I cannot easily provide code examples. I've tried to outline how to glue this all together.