0

My Drupal 7 site has been up and running for a couple years without issue. I went to log in as admin today to check for security updates and had the following experience:

  1. When I first browse to the page (as anonymous), the site works fine.
  2. Enter login information to log in as admin, immediately greeted with "Access denied".
  3. Browse other pages in the same browser session, the entire site says "Access denied".
  4. Visit the website in a new browser session, a private browser session, a private browser session on a different computer, or a private browser session on a different computer at a remote site with a different IP address, all using 4 different browsers - all say "Access denied".

The first thing the internet said was to edit settings.php and modify the $cookie_domain setting, which I had left at default (unconfigured). I tried setting this to my domain but it made no difference.

After extensive troubleshooting, I have identified the following additional behavior:

  • When I log in and get Access Denied, it only affects a particular subdomain even though it's the same site (example.com vs www.example.com).
  • When the site is in a working state, the accesslog table shows anonymous browsing with uid of 0. After I attempt to log in, the accesslog shows activity with a uid of NULL. This makes some sense if there's no session in the sessions table to match the cookie in my browser.
  • The watchdog table shows no errors, only "user Session opened for %name." and "access denied user/1".
  • Anonymous access to the site returns to normal after a CRON job runs. Since I installed Drush I can now run CRON manually which speeds up troubleshooting.
  • If I attempt to log in using correct credentials, it breaks the site again and access is denied (as described) until the next CRON job.
  • If I attempt to log in using incorrect credentials, I get "unrecognized username or password" and the site continues to work fine for anonymous browsing.

So I started digging around:

  • I found the Anonymous user's "status" in the users table was set to 0 (blocked). I set this to 1. No change.

  • I found that the row for the Anonymous user was completely missing from the users_roles table. I added this row with values 0,1 for uid and rid respectively. No change. (This row is also missing from a new install so it may not be necessary - I'm not an expert here.)

  • I verified that the role, role_permission, and node_access tables were properly populated. No change.

  • I attempted to browse to /admin/content/node-settings/rebuild but get access denied on that, too.

  • I checked the accesslog table and recent entries from today show numerous logins with a uid of NULL.

  • I truncated the sessions table and when I attempt to browse or log in to the site no new rows are added to the table (similar to this question).

  • I installed drush and cleared the cache and ran the cron logout as suggested by the answer in the StackOverflow question. This reduced the problem from everyone getting access denied to only me getting access denied, resolved by clearing the site's session cookie. So it's an improvement in that attempting to log in doesn't break the site for everyone now. Still no rows in the sessions table.

  • I hashed a new password and set it directly in the database. No change in behavior.

  • I have attempted to reset my password using the password reset links. I receive the password reset e-mail. When I open the link, I get a new accesslog entry with uid of 0 and when I click "login" I get an accesslog entry with uid of 1 (my admin user), and then when I attempt to reset my password or go anywhere else on the site, I get accesslog entries with uid of NULL. Sessions table is still empty. This works the same if I use the command drush uli to generate a login.

  • I found this question/answer and checked my themes. Unable to access the admin pages, I checked in the database and found my current theme was set to 0 (disabled). I set this to 1 (enabled) as well as its prerequisites. No change.

  • I found this question/answer and attempted a repair on my sessions table. I had to set it o use MyASM instead of InnoDB to run the repair. Repair succeeds with no change to behavior.

  • I found this question/answer and disabled all modules using the provided drush command. No change in behavior. Other suggestions in that answer either do not apply to my situation (hosted platform) or likewise do not work.

  • I attempted to install a new instance of Drupal 7 on the same server. The new instance has the exact same problem. This leads me to believe it's something systemic on the server. Based on the above StackOverflow answer I suspect my hosting provider may have forced an update from PHP 5.3 to 5.4 without notifying me. I have now tried several PHP versions available from my hosting provider (5.4, 5.5, 5.6, and 7.0) in addition to all of the steps outlined above.

I have streamlined my troubleshooting process to the following steps:

  1. Make a change
  2. Clear browser cookies
  3. Use Drush to clear cache and run CRON
  4. Refresh website and clear cookies again if a session cookies shows up
  5. Test login -> Access Denied
  6. Check Watchdog and Sessions tables in PHPMyAdmin

So far nothing seems to be making any difference at all. The majority of suggestions I've found on the internet all point to clearing cookies or setting the $cookie_domain variable which make no difference.

Can anyone help me regain admin access to my site and get this working?


More notes

  • I have created a duplicate of my site at an alternate URL for invasive testing.
  • I have attempted a new install of Drupal 7.59 using my hosting provider's CPanel. Same behavior.
  • I have attempted a new instance of Drupal 7.59 downloading directly from Drupal.com and running install.php. Same behavior.

This HAS to be something on the server itself.

At this point I'm looking at building a copy of my site on a VM at home to try and fix while I look for another hosting provider.

I'd like to migrate to Drupal 8, but that's a huge project - finding replacements for themes and modules that aren't supported in Drupal 8 and migrating data and content. I'm not looking forward to that with admin access broken.


Modules list from Drush:

 Package                      Name                                                         Type    Status         Version
 Administration               Module Missing Message Fixer (module_missing_message_fixer)  Module  Enabled        7.x-1.7
 Administration               Rename Admin Paths (rename_admin_paths)                      Module  Enabled        7.x-2.3
 Advanced CSS/JS Aggregation  AdvAgg Async Font Loader (advagg_font)                       Module  Enabled        7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg Bundler (advagg_bundler)                              Module  Enabled        7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg CDN CSS (advagg_css_cdn)                              Module  Not installed  7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg CDN Javascript (advagg_js_cdn)                        Module  Not installed  7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg Compress CSS (advagg_css_compress)                    Module  Enabled        7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg Compress Javascript (advagg_js_compress)              Module  Enabled        7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg CSS/JS Validator (advagg_validator)                   Module  Not installed  7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg External Compression (advagg_ext_compress)            Module  Disabled       7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg Modifier (advagg_mod)                                 Module  Enabled        7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg Relocate (advagg_relocate)                            Module  Not installed  7.x-2.30
 Advanced CSS/JS Aggregation  AdvAgg Subresource Integrity (advagg_sri)                    Module  Not installed  7.x-2.30
 Advanced CSS/JS Aggregation  Advanced CSS/JS Aggregation (advagg)                         Module  Enabled        7.x-2.30
 CCK                          CCK (cck)                                                    Module  Enabled        7.x-3.0-alpha3
 CCK                          Content Migrate (content_migrate)                            Module  Not installed  7.x-3.0-alpha3
 Chaos tool suite             Bulk Export (bulk_export)                                    Module  Not installed  7.x-1.12
 Chaos tool suite             Chaos tools (ctools)                                         Module  Enabled        7.x-1.12
 Chaos tool suite             Chaos Tools (CTools) AJAX Example (ctools_ajax_sample)       Module  Not installed  7.x-1.12
 Chaos tool suite             Chaos Tools (CTools) Plugin Example (ctools_plugin_example)  Module  Not installed  7.x-1.12
 Chaos tool suite             Custom content panes (ctools_custom_content)                 Module  Not installed  7.x-1.12
 Chaos tool suite             Custom rulesets (ctools_access_ruleset)                      Module  Not installed  7.x-1.12
 Chaos tool suite             Page manager (page_manager)                                  Module  Not installed  7.x-1.12
 Chaos tool suite             Stylizer (stylizer)                                          Module  Not installed  7.x-1.12
 Chaos tool suite             Term Depth access (term_depth)                               Module  Not installed  7.x-1.12
 Chaos tool suite             Views content panes (views_content)                          Module  Not installed  7.x-1.12
 Core                         Aggregator (aggregator)                                      Module  Not installed  7.59
 Core                         Block (block)                                                Module  Enabled        7.59
 Core                         Blog (blog)                                                  Module  Not installed  7.59
 Core                         Book (book)                                                  Module  Not installed  7.59
 Core                         Color (color)                                                Module  Enabled        7.59
 Core                         Comment (comment)                                            Module  Not installed  7.59
 Core                         Contact (contact)                                            Module  Disabled       7.59
 Core                         Content translation (translation)                            Module  Disabled       7.59
 Core                         Contextual links (contextual)                                Module  Disabled       7.59
 Core                         Dashboard (dashboard)                                        Module  Disabled       7.59
 Core                         Database logging (dblog)                                     Module  Enabled        7.59
 Core                         Field (field)                                                Module  Enabled        7.59
 Core                         Field SQL storage (field_sql_storage)                        Module  Enabled        7.59
 Core                         Field UI (field_ui)                                          Module  Enabled        7.59
 Core                         File (file)                                                  Module  Enabled        7.59
 Core                         Filter (filter)                                              Module  Enabled        7.59
 Core                         Forum (forum)                                                Module  Not installed  7.59
 Core                         Help (help)                                                  Module  Disabled       7.59
 Core                         Image (image)                                                Module  Enabled        7.59
 Core                         List (list)                                                  Module  Enabled        7.59
 Core                         Locale (locale)                                              Module  Enabled        7.59
 Core                         Menu (menu)                                                  Module  Enabled        7.59
 Core                         Node (node)                                                  Module  Enabled        7.59
 Core                         Number (number)                                              Module  Enabled        7.59
 Core                         OpenID (openid)                                              Module  Not installed  7.59
 Core                         Options (options)                                            Module  Enabled        7.59
 Core                         Overlay (overlay)                                            Module  Disabled       7.59
 Core                         Path (path)                                                  Module  Enabled        7.59
 Core                         PHP filter (php)                                             Module  Disabled       7.59
 Core                         Poll (poll)                                                  Module  Disabled       7.59
 Core                         RDF (rdf)                                                    Module  Enabled        7.59
 Core                         Search (search)                                              Module  Enabled        7.59
 Core                         Shortcut (shortcut)                                          Module  Disabled       7.59
 Core                         Statistics (statistics)                                      Module  Enabled        7.59
 Core                         Syslog (syslog)                                              Module  Not installed  7.59
 Core                         System (system)                                              Module  Enabled        7.59
 Core                         Taxonomy (taxonomy)                                          Module  Enabled        7.59
 Core                         Testing (simpletest)                                         Module  Not installed  7.59
 Core                         Text (text)                                                  Module  Enabled        7.59
 Core                         Toolbar (toolbar)                                            Module  Enabled        7.59
 Core                         Tracker (tracker)                                            Module  Not installed  7.59
 Core                         Trigger (trigger)                                            Module  Not installed  7.59
 Core                         Update manager (update)                                      Module  Enabled        7.59
 Core                         User (user)                                                  Module  Enabled        7.59
 Development                  Metatag: Devel (metatag_devel)                               Module  Not installed  7.x-1.22
 Feeds                        Feeds (feeds)                                                Module  Not installed  7.x-2.0-alpha8
 Feeds                        Feeds Admin UI (feeds_ui)                                    Module  Not installed  7.x-2.0-alpha8
 Feeds                        Feeds Import (feeds_import)                                  Module  Not installed  7.x-2.0-alpha8
 Feeds                        Feeds News (feeds_news)                                      Module  Not installed  7.x-2.0-alpha8
 ImageAPI                     ImageAPI (imageapi)                                          Module  Disabled       7.x-1.x-dev
 ImageAPI                     ImageAPI GD2 (imageapi_gd)                                   Module  Disabled       7.x-1.x-dev
 ImageAPI                     ImageAPI ImageMagick (imageapi_imagemagick)                  Module  Not installed  7.x-1.x-dev
 Menu                         TB Mega Menu (tb_megamenu)                                   Module  Enabled        7.x-1.0-rc2
 Multilingual -               Panels translation (i18n_panels)                             Module  Not installed  7.x-3.9
 Internationalization
 Other                        Chosen (chosen)                                              Module  Disabled       7.x-1.0-beta6
 Other                        Facebook comments (facebook_comments)                        Module  Enabled        7.x-1.0
 Other                        Fast 404 (fast_404)                                          Module  Enabled        7.x-1.5
 Other                        Font Awesome (fontawesome)                                   Module  Disabled       7.x-1.0-beta6
 Other                        Job Scheduler (job_scheduler)                                Module  Not installed  7.x-2.0-alpha3
 Other                        Job Scheduler Trigger (job_scheduler_trigger)                Module  Not installed  7.x-2.0-alpha3
 Other                        Libraries (libraries)                                        Module  Enabled        7.x-2.3
 Other                        Pathauto (pathauto)                                          Module  Enabled        7.x-1.3
 Other                        Quicktabs (quicktabs)                                        Module  Enabled        7.x-3.8
 Other                        Quicktabs Styles (quicktabs_tabstyles)                       Module  Enabled        7.x-3.8
 Other                        Site map (site_map)                                          Module  Not installed  7.x-1.3
 Other                        Special menu items (special_menu_items)                      Module  Enabled        7.x-1.0-beta6
 Other                        Token (token)                                                Module  Enabled        7.x-1.7
 Panels                       Mini panels (panels_mini)                                    Module  Enabled        7.x-3.9
 Panels                       Panel nodes (panels_node)                                    Module  Enabled        7.x-3.9
 Panels                       Panels (panels)                                              Module  Enabled        7.x-3.9
 Panels                       Panels In-Place Editor (panels_ipe)                          Module  Enabled        7.x-3.9
 Performance and scalability  APC - Alternative PHP Cache (apc)                            Module  Disabled       7.x-1.0-beta6
 Performance and scalability  HTTP Parallel Request Library (httprl)                       Module  Enabled        7.x-1.14
 Radioactivity                Radioactivity (radioactivity)                                Module  Enabled        7.x-2.10
 Radioactivity                Radioactivity defaults (radioactivitydefaults)               Module  Not installed  7.x-2.10
 Responsive                   Respond.js (respondjs)                                       Module  Enabled        7.x-1.5
 Search                       Custom Search (custom_search)                                Module  Enabled        7.x-1.20
 Search                       Custom Search Blocks (custom_search_blocks)                  Module  Not installed  7.x-1.20
 Search                       Custom Search Internationalization (custom_search_i18n)      Module  Not installed  7.x-1.20
 Search                       Custom Search Taxonomy (custom_search_taxonomy)              Module  Not installed  7.x-1.20
 SEO                          Metatag (metatag)                                            Module  Enabled        7.x-1.22
 SEO                          Metatag Importer (metatag_importer)                          Module  Not installed  7.x-1.22
 SEO                          Metatag: App Links (metatag_app_links)                       Module  Not installed  7.x-1.22
 SEO                          Metatag: Context (metatag_context)                           Module  Not installed  7.x-1.22
 SEO                          Metatag: Dublin Core (metatag_dc)                            Module  Not installed  7.x-1.22
 SEO                          Metatag: Dublin Core Advanced (metatag_dc_advanced)          Module  Not installed  7.x-1.22
 SEO                          Metatag: Facebook (metatag_facebook)                         Module  Enabled        7.x-1.22
 SEO                          Metatag: favicons (metatag_favicons)                         Module  Not installed  7.x-1.22
 SEO                          Metatag: Google Custom Search Engine (CSE)                   Module  Not installed  7.x-1.22
                              (metatag_google_cse)
 SEO                          Metatag: Google+ (metatag_google_plus)                       Module  Not installed  7.x-1.22
 SEO                          Metatag: hreflang (metatag_hreflang)                         Module  Not installed  7.x-1.22
 SEO                          Metatag: Mobile & UI Adjustments (metatag_mobile)            Module  Enabled        7.x-1.22
 SEO                          Metatag: OpenGraph (metatag_opengraph)                       Module  Enabled        7.x-1.22
 SEO                          Metatag: OpenGraph Products (metatag_opengraph_products)     Module  Not installed  7.x-1.22
 SEO                          Metatag: Panels (metatag_panels)                             Module  Not installed  7.x-1.22
 SEO                          Metatag: Twitter Cards (metatag_twitter_cards)               Module  Enabled        7.x-1.22
 SEO                          Metatag: Verification (metatag_verification)                 Module  Enabled        7.x-1.22
 SEO                          Metatag: Views (metatag_views)                               Module  Enabled        7.x-1.22
 Sharing                      ShareThis (sharethis)                                        Module  Enabled        7.x-2.13
 User interface               CKEditor (ckeditor)                                          Module  Enabled        7.x-1.18
 User interface               Gallery Formatter (galleryformatter)                         Module  Enabled        7.x-1.4
 User interface               jQuery plugins (jquery_plugin)                               Module  Enabled        7.x-1.0
 User interface               jQuery Update (jquery_update)                                Module  Enabled        7.x-2.7
 User interface               Superfish (superfish)                                        Module  Enabled        7.x-1.9
 User interface               Wysiwyg (wysiwyg)                                            Module  Enabled        7.x-2.4
 Views                        Views (views)                                                Module  Enabled        7.x-3.18
 Views                        Views Content Cache (views_content_cache)                    Module  Enabled        7.x-3.0-alpha3
 Views                        Views Slideshow (views_slideshow)                            Module  Enabled        7.x-3.9
 Views                        Views Slideshow Simple Pager (views_slideshow_simple_pager)  Module  Not installed  7.x-3.9
 Views                        Views Slideshow: Cycle (views_slideshow_cycle)               Module  Enabled        7.x-3.9
 Views                        Views UI (views_ui)                                          Module  Enabled        7.x-3.18
 XML sitemap                  XML sitemap (xmlsitemap)                                     Module  Enabled        7.x-2.3
 XML sitemap                  XML sitemap custom (xmlsitemap_custom)                       Module  Enabled        7.x-2.3
 XML sitemap                  XML sitemap engines (xmlsitemap_engines)                     Module  Enabled        7.x-2.3
 XML sitemap                  XML sitemap internationalization (xmlsitemap_i18n)           Module  Not installed  7.x-2.3
 XML sitemap                  XML sitemap menu (xmlsitemap_menu)                           Module  Enabled        7.x-2.3
 XML sitemap                  XML sitemap node (xmlsitemap_node)                           Module  Enabled        7.x-2.3
 XML sitemap                  XML sitemap taxonomy (xmlsitemap_taxonomy)                   Module  Enabled        7.x-2.3
 XML sitemap                  XML sitemap user (xmlsitemap_user)                           Module  Disabled       7.x-2.3
 Core                         Bartik (bartik)                                              Theme   Disabled       7.59
 Core                         Garland (garland)                                            Theme   Disabled       7.59
 Core                         Seven (seven)                                                Theme   Enabled        7.59
 Core                         Stark (stark)                                                Theme   Disabled       7.59
 Other                        nucleus (nucleus)                                            Theme   Enabled        7.x-1.5
 Other                        nucleus_starter (nucleus_starter)                            Theme   Enabled        7.x-1.5
 Other                        Tb Sirate (tb_sirate)                                        Theme   Enabled        7.x-1.1
  • which version of drupal 7 are you using? – Alireza Tabatabaeian Aug 22 '18 at 6:04
  • Maybe disabling APC module using drush might help you, And I think you are right about php version, has you cleared browser cookies and sessions after modifying php version? – Alireza Tabatabaeian Aug 22 '18 at 6:20
  • I was using 7.5, but while working with my hosting provider they pushed 7.59. I have created a copy of the site for troubleshooting. I've tried disabling ALL modules with no change (flushing caches, using a private browser session to avoid cache nonsense. And again it does it even with a clean new instance of Drupal 7.59, so it can't be a module it has to be something on the server. – Thomas Aug 22 '18 at 17:10
0

This was finally resolved by my web host. Good grief it was frustrating!

I still do not know for sure what the solution was. They had a widespread (but publicly unidentified) database issue that was causing a number of problems which may have been the solution. Unfortunately they did not notify me when that issue was resolved so I did not test until after working with them to resolve a different problem related to a php.ini conflict.

Sorry that this answer can't be more helpful, but if you have the same problems then at least you can be confident it's something with your host. If you self-host, I recommend rebuilding a server from the ground up at least as a test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.