2

I am working on a website which has around 150k users. I have to create a csv file which contains the details of user (drupal fields and profile2 fields ) and export the csv to another server daily.

I have implemented by export function in a hook_cron and in the function i do a user_load function call and two profile2_load_by_user and some other custom mysql queries.

The problem which i have is that my export_function does not seem to terminate, it may be a timeout or a memory problem.

I have tested my function my limiting the number of users to around 1000 and it works fine.

I am wondering whether it is possible to export huge volume of data through hook_cron.

1

Why don't you do this in chunks.I mean schedule you cron using elysia_cron or ultimate cron at every hour or something and export chunks of users at a time,flag users you exported and at every next cron run get the new ones.

One important thing you should have crontab set on your server,do not use drupal's default poormanscron.

1

I work with large data sets in the form of Webform results. To export 60k records with 30+ columns of data I had to kick the memory limit up in my .htaccess file.

# FILE: .htaccess
# PHP 5, Apache 1 and 2.
<IfModule mod_php5.c>
  php_value magic_quo...
  ...
  # 2013-09-30 -- These new settings have been put in place so that the data dump (download) for the webform module doesn't exceed memory limits causing it to crash. -- 512 to 1024
  php_value memory_limit 1024M
  php_value max_execution...
</IfModule>

you may also want to change your php_value max_execution_time from 180 to something a little higher if your request is timing out.

0

I would suggest creating a drush command that retrieves and saves the data via batch or queue. This article may come handy which provides the following code examples:

function mymodule_setup_batch($start=1, $stop=100000) {
  //  ...
  //  Populate $lots_of_data from record $start to record $stop.
  //  ...

  //Break up all of our data so each process does not time out.
  $chunks = array_chunk($lots_of_data, 20);
  $operations = array();
  $count_chunks = count($chunks);

  //for every chunk, assign some method to run on that chunk of data
  foreach ($chunks as $chunk) {
    $i++;
    $operations[] = array("mymodule_method_to_work_on_a_small_part", array( $chunk ,'details'=> t('(Importing chunk @chunk  of  @count)', array('@chunk '=>$i, '@count'=>$count_chunks))));
    $operations[] = array("mymodule_another_method",array($chunk));
  }

  //put all that information into our batch array
  $batch = array(
    'operations' => $operations,
    'title' => t('Import batch'),
    'init_message' => t('Initializing'),
    'error_message' => t('An error occurred'),
    'finished' => 'mymodule_finished_method'
  );

  //Get the batch process all ready!
  batch_set($batch);
  $batch =& batch_get();

  //Because we are doing this on the back-end, we set progressive to false.
  $batch['progressive'] = FALSE;

  //Start processing the batch operations.
  drush_backend_batch_process();
}

function mymodule_method_to_work_on_a_small_part ($chunk, $operation_details, &$context) {
  //Do something to $chunk, maybe create a node?
  $context['message'] = $operation_details; //Will show what chunk we're on.
}
function mymodule_another_method($chunk, &$context) {
  //Do some more work.
  $context['message'] = t('We have done a second thing to a chunk of data');
}

function mymodule_finished_method($success, $results, $operations) {
  //Let the user know we have finished!
  print t('Finished importing!');
}

If you like, you can implement Drush command, e.g.

function mymodule_drush_command() {
  $items  = array();
  $items['myimport'] = array(
    'callback'    => 'mymodule_setup_batch',
    'description' => dt('Import'),
    'arguments'   => array(
      'start'     => "start",
      'stop'      => "stop",
    ),
  );
  return $items;
}

function mymodule_drush_help($section) {
  switch ($section) {
    case 'drush:myimport':
      return dt("import items from the Internal Database [start record] [end record].");
  }
}
0

In Drupal 8, you can use Default Content module to export users into CSV file via drush command which can be invoked periodically by adding into the system's crontab. It's better to approach, as having big chunk of data exports in hook_cron, as it's blocking the other important processing.


Another way is to create a View which exports users in CSV format by using Views Datasource module. Then you can enable caching, so the results are cached.


As the suggestion, to avoid using too many resources on processing and memory (such as a huge number of calls to user_load), you should implement memory caching service such as Redis or Memcached. Further more, please check: How do you improve Drupal performance?


If you still got the issue, use PHP profiler (such as XDebug) to find the cause of your issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.