No one can login and access denied message

Question

The site worked till yesterday and just now it's behaving strangely.

• The site is not created with features, an archive dump including the code and database is copied to server, and the old installation is replaced with the new one manually (unfortunately). Now it is irrelavant also happening on local server.
• No caching module is installed. Session API is not also installed.
• Theme is set to 'seven' (core admin theme, just in case).
• The watchdog does not contain any error, warning or info except the usual "session opened for user xyz";
• The settings file does not contain any special cookie directive. It never did.
• The site is running on php 5.6, Apache 2.2, CentOS 6. PHP is running as fast-cgi. Same as always.
• It's Drupal 7.

I go to example.com/user/login, enter username and password, the form is submitted successfully, the page is redirected to example.com/user/%uid (tried both uid 1 and other users). and boom! I get an access denied. And the user is not logged in.

If I truncate the sessions table with truncate table sessions I can log in once and once only, with any user. Then again no one can log-in not even uid 1.

Also with a password reset link (using the command drush uli I can log in with no problem).

I disabled all contrib modules and theme, but the same problem persists.

What is causing such problems with sessions? any ideas?

Answer 1

The problem was ip_geoloc module. It was messing up $_SESSION variable, a solution is given on the issue queue (for now!) and the module author agrees with it. It's highly probable that it is happening because of a mis-configuration by me. The user was successfully logged in but immediately their session was invalidated. Somehow disabling it was not enough and it's directory had to be moved out of Drupal root. I am 100% sure not any of it's functions or APIs are used anywhere else. Maybe some cache problem?

Lesson learned:

When having signing-in problem, try to check the $_SESSION variable on the server using devel and it's helper functions (e.g dpm, dsm, dd, ... or as a last resort var_export) and not with you google/firefox developers console.

This is the third time I'm facing this strange behavior and the first time I was able to fix. If you are having this problem, try to follow this checklist:

Ask yourself:

• Does truncating the sessions table fix the problem?
• Does forcing a change for user password fix the problem?
• Have you added any particular configuration to your settings.php? e.g base_url, cookie_domain, ...
• Does UID 0 exist in your database (i.e anonymous user)?
• Is any redirection module installed? search404, redirect, global_redirect,...
• Do you have a custom module enabled?
• Is there any problems with your theme?
• Do you have any path altering modules, e.g path_alias?
• Do you have any module changing the log-in behavior? e.g logintoboggan
• Have you enabled any caching module? e.g authcache, memcache, session_cache, boost, varnish, ...
• Do you have any access control module enabled? e.g workbench, content_access, acl, og_access, ...
• And finally, have you enabled any module which changes the user $_SESSION? e.g session_api, session_cache, ip_geoloc, ...

Places to look / Things to do

• Look at drupals watchdog messages. If you can not log-in, use drush watchdog-show --tail to see what's happening.
• If running apache and mod-php, look in apache's error log, usually at /var/log/apache/error_log for *nix systems.
• If running php-fpm, see the php-fpm log, make sure there is no communication problem with php daemon, such as headers are not lost. if you see no error log try running your site with drush/php internal server. the command is drush rs or drush rs the.ip.goes.here:thePort
• make a fullbackup, disable all modules with this command AFTER making a full backup (you can make a backup with drush ard or just database drush sql-dump > db.sql or use backup_migrate module)
  drush dis -y $(drush pml --no-core --status=enabled --pipe) -> this command only works in *nix systems I don't know the direct translation for windows.
• Set your default theme to something... safe! such as seven or garland. If you can not log-in use this drush commands:
  drush en seven && drush vset theme_default seven;
• Monitor $_SESSION variable with any debugging tool, see if it's content are correct, ok, healthy.
• For all the processes and daemons involved in serving a request (nginx, php-fpm, apache, varnish...) make sure they all have write access to their tmp directory.

Good luck debugging!

Answer 2

What worked for me was enabling and configuring the $cookie_domain variable in settings.php (because I have a site that can be accessed by 2 different domains).

Pay attention to the comment before this variable! The domain name must start with a period. EX:

$cookie_domain = '.example.com';

Good luck!