Logging into Drupal 7 site with headless browser: why would it fail?

Question

NOTE: I've heavily revised the wording of my question to focus more clearly on the problem. There's a ton of really really useful information in the answers below, but not all of it applies to solving the problem of logging in with a headless browser. (Feel free to view the edit history for more details, as well as a more verbose version of the same info.)

I'm using a headless WebKit browser (from CasperJS to log into my Drupal 7 site, and it is failing for unknown reasons. I first used the Ruby version of the Mechanize library and got identical results: access denied even after filling out and submitting the log in form. I'm pretty sure the problem has to do with Drupal, and some failure on my part to jump through some hoop (that is probably there to prevent spambots and such from submitting to Drupal-based sites).

Since my script loads the log-in form before submitting it, AFAIK it should work just like a normal user logging in. I've looked through the methods mentioned in the answers that details Drupal's form-submission process, but I'm not seeing what would cause my problem.

This question seems to provide clues, though it is D6 and using curl. That person is passing field_source, form_token, form_build_id, and form_id as fields. In my form I only see form_build_id and form_id. Should I be passing all those fields, even though they don't seem to be present in the login form? Drupal seems to look for form_token, but it's not present on my login form. What's up with that?

NOTE: Someone will undoubtedly ask why I'm using these tools when there must be better ways with such-and-such a Drupal module, and the short answer is just that I chose to use tools I'm most familiar with. If you have good module suggestions please leave comments (with implementation details if possible) as I'd love to hear the suggestions, but I'd still appreciate an answer to this question as it is asked. Thanks.

Answer 1

Is there some security feature built into Drupal to prevent submissions by scripts?

Yes, there is.

In order to prevent automated form submission by bots etc.. During form build Drupal generates a token, based on session id, and a unique id, for form cache. These are used during form validation.

See:

• drupal_prepare_form
• drupal_get_token
• drupal_validate_form
• form_get_cache

In order to successfully submit the form, you would need to retrieve it and then submit it using the same token and form_build_id values.

However, there are other ways to submit forms programmatically, such as

drupal_form_submit

As well as purely code based methods to log a user in.

e.g, in a very simple form:

// Load a user account and assign to global $user
global $user;
$account = user_load(1, TRUE); // Root user
$user = $account; // This "logs" the user in
drupal_session_regenerate();
// Invoke login event
$edit = NULL;
user_module_invoke('login', $edit, $user);

Answer 2

Is there some security feature built into Drupal to prevent submissions by scripts?

Yes, there are. First off, a script that is not bootstrapping Drupal cannot submit a form that has not requested to a Drupal site. Drupal explicitly check that condition is verified, and that task is done from drupal_validate_form(), which contains the following code.

  // If the session token was set by drupal_prepare_form(), ensure that it
  // matches the current user's session.
  if (isset($form['#token'])) {
    if (!drupal_valid_token($form_state['values']['form_token'], $form['#token'])) {
      $path = current_path();
      $query = drupal_get_query_parameters();
      $url = url($path, array('query' => $query));

      // Setting this error will cause the form to fail validation.
      form_set_error('form_token', t('The form has become outdated. Copy any unsaved work in the form below and then <a href="@link">reload this page</a>.', array('@link' => $url)));
    }
  }

_form_validate() then verifies there is a correspondence between the values submitted and the form elements contained in the form requested to Drupal. If there isn't an exact correspondence, the submission fails.

  if (isset($elements['#maxlength']) && drupal_strlen($elements['#value']) > $elements['#maxlength']) {
    form_error($elements, $t('!name cannot be longer than %max characters but is currently %length characters long.', array('!name' => empty($elements['#title']) ? $elements['#parents'][0] : $elements['#title'], '%max' => $elements['#maxlength'], '%length' => drupal_strlen($elements['#value']))));
  }

        if (!isset($options[$v])) {
          form_error($elements, $t('An illegal choice has been detected. Please contact the site administrator.'));
          watchdog('form', 'Illegal choice %choice in !name element.', array('%choice' => $v, '!name' => empty($elements['#title']) ? $elements['#parents'][0] : $elements['#title']), WATCHDOG_ERROR);
        }

   elseif (!isset($options[$elements['#value']])) {
      form_error($elements, $t('An illegal choice has been detected. Please contact the site administrator.'));
      watchdog('form', 'Illegal choice %choice in %name element.', array('%choice' => $elements['#value'], '%name' => empty($elements['#title']) ? $elements['#parents'][0] : $elements['#title']), WATCHDOG_ERROR);
    }

Is there some other reason scripts would fail to log into Drupal on a page where a user in a normal browser would have no problem?

As you are asking for a headless browser, the form submission could be successful if:

• The browser requests the login form
• It inject JavaScript code that fills out the necessary fields, and submit the form back to Drupal

The condition necessary for the operation to work is that the browser is able to keep the session open. In fact, the token used to validate the form is obtained from drupal_get_token(), which contains the following code.

  return drupal_hmac_base64($value, session_id() . drupal_get_private_key() . drupal_get_hash_salt());

The function used to validate the token is drupal_valid_token().

function drupal_valid_token($token, $value = '', $skip_anonymous = FALSE) {
  global $user;
  return (($skip_anonymous && $user->uid == 0) || ($token == drupal_get_token($value)));
}

As side note, to create content is not necessary to log in as a specific user: Just bootstrap Drupal, build a node object, and call node_save().

For example, the following code put in a file in the same directory containing the index.php Drupal comes with would create a node. (The code is for Drupal 7, but a similar code can be written for Drupal 6.)

  define('DRUPAL_ROOT', getcwd());
  require_once DRUPAL_ROOT . '/includes/bootstrap.inc';
  drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL); 

  $node = new stdClass();
  $node->title = $node_title;
  $node->type = $node_type;

  // Sets some defaults. Invokes hook_prepare() and hook_node_prepare().
  node_object_prepare($node); 

  $node->language = LANGUAGE_NONE;
  $node->uid = $user_id;
  $node->status = 1;
  $node->promote = 0;

  // Comments are disabled; change it to COMMENT_NODE_OPEN to enable them.
  $node->comment = COMMENT_NODE_CLOSED;

  $node = node_submit($node);
  node_save($node);

Using this code, Drupal would create a node.

If the code is used for a publicaly available site, and you want to avoid everybody runs the file (let's suppose the file is migration.php), you can use the following code.

define('DRUPAL_ROOT', getcwd());
require_once DRUPAL_ROOT . '/includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL); 

if (!isset($_GET['cron_key']) || variable_get('migration_key', 'drupal') != $_GET['cron_key']) {
  watchdog('migration', 'Migration could not run because an invalid key was used.', array(), WATCHDOG_NOTICE);
  drupal_access_denied();
}
elseif (variable_get('maintenance_mode', 0)) {
  watchdog('Migration', 'Migration could not run because the site is in maintenance mode.', array(), WATCHDOG_NOTICE);
  drupal_access_denied();
}
else {
  $node = new stdClass();
  $node->title = $node_title;
  $node->type = $node_type;

  // Sets some defaults. Invokes hook_prepare() and hook_node_prepare().
  node_object_prepare($node); 

  $node->language = LANGUAGE_NONE;
  $node->uid = $user_id;
  $node->status = 1;
  $node->promote = 0;

  // Comments are disabled; change it to COMMENT_NODE_OPEN to enable them.
  $node->comment = COMMENT_NODE_CLOSED;

  $node = node_submit($node);
  node_save($node);
} 

The Drupal variable migration_cron should be initialized with variable_set('migration_key', drupal_get_token('update'));. The script would be then called as http://example.com/migration.php?migration_key=XYX, where XYZ is the value saved in the Drupal variable migration_key. As drupal_get_token() returns a different value each time the function is called, external attachers could not get the value to use to run the script.

The code I shown creates just a node, but using similar code would be possible to migrate content. The code would execute a loop to create a node for each entry loaded from a file, an external database, or returned from a drupal_http_request() call to an external site (among other methods). In any case, it is not necessary to first log in to create a node, if the PHP code first bootstrap code.

If you can use Drush, that would probably be a better way to programmatically create content.

As alternative, there are many modules to migrate content. As far as I recall, also the Feeds module is used to migrate content to Drupal.

Answer 3

Try this with CasperJs: Need CasperJs and PhantomJs on your system PATH

#filename testCasper.js

var _adminUser = '';
var _adminPass = '';
var _baseUrl = 'local.drupal.dev';

casper.test.begin('Basic BDD: login as Admin and test behavior of interface.', 3, function suite(test) {

casper.start(_baseUrl , function() {
    test.assertTitle("drupalLocal.dev", "homepage title is ok");
    test.assertExists('form#user-login-form', "user login form is found");
    this.fill('form#user-login-form', {
        name: _adminUser,
        pass: _adminPass
    }, true);
});

casper.then(function() {
    test.assertTextExists('Log out', 'User is logged in');
    console.log('location is ' + this.getCurrentUrl());        
});


casper.run(function() {
    test.done();
    phantom.exit();
});
});

run with:

`casperjs test testCasper.js`