How to avoid max execution timeout when importing a lot of nodes programmatically?

Question

I am creating a content synchronisation module. I have a UI that allows a user to import nodes from a remote site. They enter the site url and web service endpoint, along with some other configuration options, and the program will go and fetch the nodes and import them into the database using node_save.

I received a max timeout error while importing 357 nodes:

Fatal error: Maximum execution time of 30 seconds exceeded in D:\xampp\htdocs\drupal-7.15\modules\pull_content\pull_content.module on line 1188

So, I increased the max execution time to 360 but after 360s it timed out and I got the same error. I know that the processing hasn't frozen because I can see the nodes going into the node table at a rate of about 3 or 4 every 10s.

I don't want to keep endlessly increasing the timeout as there is the possibility that I may have to import 1000 nodes or more at one time.

Is there any way to avoid this error message other than increasing the max execution time? Maybe by running things in a batch? Or should I just make the max execution time really large?

Answer 1

Yes, there is a way! It's built into Drupal core, and it's there for this exact reason and purpose. You need to use the Batch API. Batch lets you, well, perform operations in batches across page loads. It automates the carrying of information between these page loads and helps you to show a nice progress bar to the user.

See:

• Batch operations reference
• The Examples module (which contains a batch_example module)
• Countless online tutorials

Answer 2

Normally, I use

set_time_limit(0);

somewhere in my settings.php file or my custom import script.

See set_time_limit() and max_execution_time for reference.

You can also look into the Batch API, but I don't have a ton of experience with it.