First page load/execution slow, login slower Drupal 7

Question

Drupal 7 site experiencing slow execution/load time for first page visit and slower execution/load time for login. The web server and database servers are dedicated and well resourced, the site has no traffic currently.

Settings already in place

• JS/CSS compression enabled
• Cron is run via crontab every hour and disabled in Drupal
• Cache warming 2 times a day with min, max cache expiry time at 1 day for anon
• APC is set to 192MB and using about half memory
• Database is on another dedicated box lots of RAM
• Update manager disabled
• Content translation disabled
• No missing modules

Setup info

• About 100 modules: 70 contrib, 30 custom
• Drupal core and modules are on NFS server
• apc.stat=1 , apc.stat_ctime=Off, apc.ttl=3600
• php.ini - realpath_cache_size=512k
• settings.php - database set via IP (not hostname)
• Mysql - query_cache_size no effect with both enabled or disabled
• Mysql - innodb_flush_log_at_trx_commit =2
• Mysql - innodb_buffer_pool_size=4G #current DB is 100mb total

DB queries are < 110 ms. XHprof snapshots below.

Update1: Login after few silent hours

Update2: Update.php XHprof timing

Update: NFS info

%nfsstat -m Flags: rw,nosuid,nodev,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=xx.xx.xx.xx,mountvers=3,mountport=xxxx,mountproto=udp,local_lock=none,addr=xx.xx.xx.xx

Any suggestions to improve load times?

Update 3: Cache configurations in settings.php When I enable and run test for cache and APC cache using Drupal's Testing module. I get fatal errors with "Class 'Database' not found"

I am using APC cache module, below are my settings. Is something wrong with this config. APC has 192MB and used about 110MB

/****
 * APC Caching.
 ****/
$conf['cache_backends'][] = 'sites/XYZ/modules/apc/drupal_apc_cache.inc';

// cache - is used for all common cache purposes. is a default cache bin for Drupal.
$conf['cache_class_cache'] = 'DrupalAPCCache';

$conf['cache_class_cache_admin_menu'] = 'DrupalAPCCache';

// cache_block - is used for storing block's cache
$conf['cache_class_cache_block'] = 'DrupalAPCCache';

$conf['cache_class_cache_bootstrap'] = 'DrupalAPCCache';

$conf['cache_class_cache_field'] = 'DrupalAPCCache';

// cache_filter - is used for storing filtered input
$conf['cache_class_cache_filter'] = 'DrupalAPCCache';

// cache_form - is used for storing forms. 
// This cache bin should not be assigned to volatile storage
//$conf['cache_class_cache_form'] = 'DrupalAPCCache'; // use Database

//$conf['cache_class_cache_image'] = 'DrupalAPCCache';

$conf['cache_class_cache_libraries'] = 'DrupalAPCCache';

// cache_menu - is used for storing assembled and processed menu
$conf['cache_class_cache_menu'] = 'DrupalAPCCache';

// cache_page - is used to store cached pages 
//$conf['cache_class_cache_page'] = 'DrupalAPCCache';

$conf['cache_class_cache_path'] = 'DrupalAPCCache';

$conf['cache_class_cache_path_breadcrumbs'] = 'DrupalAPCCache';

// cache_update - if module Update status' is enabled this 
// bin stores Drupal, modules and themes update information. 
$conf['cache_class_cache_update'] = 'DrupalAPCCache';


//$conf['apc_show_debug'] = TRUE;  // Remove the slashes to use debug mode.

/*****
 * Contributed module cache
 ****/

// Browscap module
$conf['cache_class_cache_browscap'] = 'DrupalAPCCache';

// Entity cache module
// This will cache items in Database when not pointed to another cache
// There is no way to disable this caching for particular entity, it is 
// all or nothing, erasing the potential little benefit for enabling this 
// module for caching entity_user to increase performance of web services
/*
//$conf['cache_class_cache_entity_comment'] = 'DrupalAPCCache';
//$conf['cache_class_cache_entity_file'] = 'DrupalAPCCache';
//$conf['cache_class_cache_entity_node'] = 'DrupalAPCCache';
//$conf['cache_class_cache_entity_taxonomy_term'] = 'DrupalAPCCache';
//$conf['cache_class_cache_entity_taxonomy_vocabulary'] = 'DrupalAPCCache';
$conf['cache_class_cache_entity_user'] = 'DrupalAPCCache';
*/

//Panels module
$conf['cache_class_cache_panels'] = 'DrupalAPCCache';

// Registration module
//$conf['cache_class_cache_entity_registration'] = 'DrupalAPCCache'; 
//$conf['cache_class_cache_entity_state'] = 'DrupalAPCCache'; 
//$conf['cache_class_cache_entity_type'] = 'DrupalAPCCache'; 

// Rules module
$conf['cache_class_rules'] = 'DrupalAPCCache';

// Token module
$conf['cache_class_cache_token'] = 'DrupalAPCCache';

// View module
$conf['cache_class_cache_views'] = 'DrupalAPCCache';
//$conf['cache_class_cache_views_data'] = 'DrupalAPCCache';

// Default DB for the ones that change too frequently and are small
$conf['cache_default_class']    = 'DrupalDatabaseCache';
// THIS MUST BE SERVED FROM DB FOR STABILITY
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';

Answer 1

If using NFS I highly recommend using apc.stat=0 and then if you want to update the apc cache run apc_clear_cache('opcode');.

Seeing how drupal_parse_info_file() is ran on every page load I'm guessing this core patch will help you out: https://www.drupal.org/node/1081266#comment-9411015

Another thing that can help with running code from a NFS drive is this issue: https://www.drupal.org/node/1443308

Also related to potential NFS issues these 2 modules usually help in that situation
https://www.drupal.org/project/imageinfo_cache
https://www.drupal.org/project/advagg

Answer 2

It seems your top time consuming function is ob_flush() which is called only once, but for 200-500ms.

This call is invoked in drupal_page_footer() as follow:

function drupal_page_footer() {
  // ...

  if (variable_get('cache', 0) && ($cache = drupal_page_set_cache())) {
    drupal_serve_page_from_cache($cache);
  }
  else {
    ob_flush();
  }

So in both of your cases cache doesn't work, either by the 1st load or for the reason that the page was loaded by authenticated user (drupal_page_is_cacheable()). So probably nothing that we can do here.

Function ob_flush() send the contents of the output buffer to the client, and the latency of this transport to the client is probably higher than the actually PHP buffer flushing. See: Why does calling ob_flush() slow down my latency so much?.

Here are few ideas how to improve the load times:

• enable/disable zlib.output_compression (and see which option is quicker),
• test for network bottlenecks, if the problem is not with NFS server by checking the load times of simple PHP files (such as update.php or plain php files),
• if your buffer is flushed too slow, something holds the buffer until the page has finished loading, this can be caused by some Apache modules or client it-self,
  • if it's server fault, review your Apache modules and APC settings (you can use strace to investigate the problem by debugging your Apache process when it handles the request),
  • if it's client fault, test your page by disabling plugins or use different web browser, as well as disable any antivirus softwares (which can hold the buffer until it's checked for web viruses)
• install XCache instead of APC, set xcache.count to your cpu count something between 4-8 (check sample xcache.ini).