You can break up your work somehow. You shouldn't try to process thousands of things at a time and assume nothing will break or you'll run out of time.
I have recently wrote an in-house Drupal 7 project that allows me to query a webservice for XML data very quickly and process 100,000s of XML entries to an SQL DB pretty fast. A full fetch of all remote SOAP based data is ~100,000 SQL rows after import. This takes ~5 mins to complete. From CURL to DB insert.
The Drupal 7 site performs the following:
- Allows me to fetch a .NET SOAP response for X departments from my university. X any number of department queries from 1 department to 90 departments per school quarter.
- I used the Background Process module to fetch the CURL requests to local XML files in a Drupal private filesystem.
- 90 CURL requests with ~2Mb XML responses complete in ~1 minute. (I could request 1 huge xml file of all departments; I break it up for many threads below).
- The XML files are added to a Drupal queue. Queue's can lease queue items as jobs for processing. A developer can set a lease time per job for any length of time.
- I then use hook_cron_queue to define a worker function that parses the XML and inserts the data into a database. This happens to be a non-Drupal database that I connect to via a custom db entry in
settings.php.
- I use the Ultimate Cron and Ultimate Cron Queue Scaler modules to actually perform the processing. I set 9 threads in Ultimate Cron Queue to process queue items, and I have set my lease time to literally 2 hours (7600 seconds). Drupal cron runs/checks for work every minute, the lease time assures no thread tramples over work already in progress.
- with no optimizations a script I'd written using PHP
DomDocument with Xpath queries inserting SQL queries to a DB took minutes per XML file. After adding some SQL batch inserts with db_insert each file takes seconds, but I know from progressively optimizing my scripts a job can have a set lease time. Which is what you want here.
In short. Find a way to break-up your workload and dont be afraid to use watchdog log messages in your module code. A Queue is a nice way to store pieces of the whole task to complete in tiny parts. As you run your worker functions you'll be able to find a lease time that works best for your data size and processing times.
I realized your post is about Drupal 6; it doesn't really matter. The core pieces you want are the D6 queue project and hook_cron(). The Ultimate Cron, Ultimate Cron Queue Scaler, and the Background Process modules have D6 versions.
The project described is a private in-house project, and I cannot easily provide code examples. I've tried to outline how to glue this all together.
