NOTE: I've heavily revised the wording of my question to focus more clearly on the problem. There's a ton of really really useful information in the answers below, but not all of it applies to solving the problem of logging in with a headless browser. (Feel free to view the edit history for more details, as well as a more verbose version of the same info.)

I'm using a headless WebKit browser (from CasperJS to log into my Drupal 7 site, and it is failing for unknown reasons. I first used the Ruby version of the Mechanize library and got identical results: access denied even after filling out and submitting the log in form. I'm pretty sure the problem has to do with Drupal, and some failure on my part to jump through some hoop (that is probably there to prevent spambots and such from submitting to Drupal-based sites).

Since my script loads the log-in form before submitting it, AFAIK it should work just like a normal user logging in. I've looked through the methods mentioned in the answers that details Drupal's form-submission process, but I'm not seeing what would cause my problem.

This question seems to provide clues, though it is D6 and using curl. That person is passing field_source, form_token, form_build_id, and form_id as fields. In my form I only see form_build_id and form_id. Should I be passing all those fields, even though they don't seem to be present in the login form? Drupal seems to look for form_token, but it's not present on my login form. What's up with that?

NOTE: Someone will undoubtedly ask why I'm using these tools when there must be better ways with such-and-such a Drupal module, and the short answer is just that I chose to use tools I'm most familiar with. If you have good module suggestions please leave comments (with implementation details if possible) as I'd love to hear the suggestions, but I'd still appreciate an answer to this question as it is asked. Thanks.

NOTE: I've heavily revised the wording of my question to focus more clearly on the problem. There's a ton of really really useful information in the answers below, but not all of it applies to solving the problem of logging in with a headless browser. (Feel free to view the edit history for more details, as well as a more verbose version of the same info.)

I'm using a headless WebKit browser (from CasperJS to log into my Drupal 7 site, and it is failing for unknown reasons. I first used the Ruby version of the Mechanize library and got identical results: access denied even after filling out and submitting the log in form. I'm pretty sure the problem has to do with Drupal, and some failure on my part to jump through some hoop (that is probably there to prevent spambots and such from submitting to Drupal-based sites).

Since my script loads the log-in form before submitting it, AFAIK it should work just like a normal user logging in. I've looked through the methods mentioned in the answers that details Drupal's form-submission process, but I'm not seeing what would cause my problem.

This question seems to provide clues, though it is D6 and using curl. That person is passing field_source, form_token, form_build_id, and form_id as fields. In my form I only see form_build_id and form_id. Should I be passing all those fields, even though they don't seem to be present in the login form? Drupal seems to look for form_token, but it's not present on my login form. What's up with that?

NOTE: Someone will undoubtedly ask why I'm using these tools when there must be better ways with such-and-such a Drupal module, and the short answer is just that I chose to use tools I'm most familiar with. If you have good module suggestions please leave comments (with implementation details if possible) as I'd love to hear the suggestions, but I'd still appreciate an answer to this question as it is asked. Thanks.

NOTE: I've heavily revised the wording of my question to focus more clearly on the problem. There's a ton of really really useful information in the answers below, but not all of it applies to solving the problem of logging in with a headless browser. (Feel free to view the edit history for more details, as well as a more verbose version of the same info.)

I'm using a headless WebKit browser (from CasperJS to log into my Drupal 7 site, and it is failing for unknown reasons. I first used the Ruby version of the Mechanize library and got identical results: access denied even after filling out and submitting the log in form. I'm pretty sure the problem has to do with Drupal, and some failure on my part to jump through some hoop (that is probably there to prevent spambots and such from submitting to Drupal-based sites).

Since my script loads the log-in form before submitting it, AFAIK it should work just like a normal user logging in. I've looked through the methods mentioned in the answers that details Drupal's form-submission process, but I'm not seeing what would cause my problem.

NOTE: Someone will undoubtedly ask why I'm using these tools when there must be better ways with such-and-such a Drupal module, and the short answer is just that I chose to use tools I'm most familiar with. If you have good module suggestions please leave comments (with implementation details if possible) as I'd love to hear the suggestions, but I'd still appreciate an answer to this question as it is asked. Thanks.

NOTE: I've heavily revised the wording of my question to focus more clearly on the problem. There's a ton of really really useful information in the answers below, but not all of it applies to solving the problem of logging in with a headless browser. (Feel free to view the edit history for more details, as well as a more verbose version of the same info.)

I'm using a headless WebKit browser (from CasperJS to log into my Drupal 7 site, and it is failing for unknown reasons. I first used the Ruby version of the Mechanize library and got identical results: access denied even after filling out and submitting the log in form. I'm pretty sure the problem has to do with Drupal, and some failure on my part to jump through some hoop (that is probably there to prevent spambots and such from submitting to Drupal-based sites).

Since my script loads the log-in form before submitting it, AFAIK it should work just like a normal user logging in. I've looked through the methods mentioned in the answers that details Drupal's form-submission process, but I'm not seeing what would cause my problem.

This question seems to provide clues, though it is D6 and using curl. That person is passing field_source, form_token, form_build_id, and form_id as fields. In my form I only see form_build_id and form_id. Should I be passing all those fields, even though they don't seem to be present in the login form? Drupal seems to look for form_token, but it's not present on my login form. What's up with that?

NOTE: Someone will undoubtedly ask why I'm using these tools when there must be better ways with such-and-such a Drupal module, and the short answer is just that I chose to use tools I'm most familiar with. If you have good module suggestions please leave comments (with implementation details if possible) as I'd love to hear the suggestions, but I'd still appreciate an answer to this question as it is asked. Thanks.

I've written 2 scripts (in 2 different languages) to automatically log in to my Drupal site, in order to post content which I'm migrating from an old site.

I wrote the first version in Ruby using the Mechanize library (similar the the library of the same name for Perl, and there's another for Python), and the second in CasperJS. Both gave the same result: everything worked fine, up until the login. After (seemingly) logging in, I got access denied errors.

At first (when this happened with the Ruby version) I assumed it was because Drupal was setting session cookies with JavaScript. (One limitation of the Mechanize library is that it does not execute JavaScript, so if session cookies are set by JavaScript, you're hosed.) So I tried doing the same thing with CasperJS since (besides being another very nice and powerful tool) it does execute JavaScript.

When I got the exact same result with JavaScript and CasperJS, I actually bothered to look to see if Drupal is setting cookies in JavaScript and could find no evidence that it was doing that. So it seems the problem is somehow with Drupal, but I don't know precisely what.

It seems that the 'form_build_id' field is for some reason not getting set properly by the server when serving to the script, or is not getting submitted properly, but I don't know why. So the core of the question: Is there some security feature built into Drupal to prevent submissions by scripts? If so, would it apply to these scripts (which are actually driving a headless browser!)? Is there some other reason scripts would fail to log into Drupal on a page where a user in a normal browser would have no problem?

NOTE: Someone will undoubtedly ask why I'm using these tools when there must be better ways with such-and-such a Drupal module, and the short answer is just that I chose to use tools I'm most familiar with. If you have good module suggestions please leave comments (with implementation details if possible) as I'd love to hear the suggestions, but I'd still appreciate an answer to this question as it is asked. Thanks.

NOTE: I've heavily revised the wording of my question to focus more clearly on the problem. There's a ton of really really useful information in the answers below, but not all of it applies to solving the problem of logging in with a headless browser. (Feel free to view the edit history for more details, as well as a more verbose version of the same info.)

I'm using a headless WebKit browser (from CasperJS to log into my Drupal 7 site, and it is failing for unknown reasons. I first used the Ruby version of the Mechanize library and got identical results: access denied even after filling out and submitting the log in form. I'm pretty sure the problem has to do with Drupal, and some failure on my part to jump through some hoop (that is probably there to prevent spambots and such from submitting to Drupal-based sites).

Since my script loads the log-in form before submitting it, AFAIK it should work just like a normal user logging in. I've looked through the methods mentioned in the answers that details Drupal's form-submission process, but I'm not seeing what would cause my problem.

NOTE: Someone will undoubtedly ask why I'm using these tools when there must be better ways with such-and-such a Drupal module, and the short answer is just that I chose to use tools I'm most familiar with. If you have good module suggestions please leave comments (with implementation details if possible) as I'd love to hear the suggestions, but I'd still appreciate an answer to this question as it is asked. Thanks.