Advices for tunning my server for better performance and scalability

Question

I am working on a web application which will primarily be used by authenticated users. The web application imports records from CSV files using the Feeds module, MongoDB was chosen to store this data because it is much faster to do the writes rather than in MySQL. The application is importing thousands of records every day really fast. The records are referenced to the users and when they log in, they can consult their information. The application is also used as an API for the mobile app.

I have a 17gb AWS EC2 Instance as the server.

Varnish is install working perfectly serving content for anonymous users. I'm getting around 3000-4000 requests per second which is more than enough.

My problem is when benchmarking for authenticated users, on API pages that are only json content I get only 24 requests per second. And other pages I get like 9 - 11 requests per second. Here is some apache benchmark output.

Concurrency Level:      100
Time taken for tests:   9.154 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      1790900 bytes
HTML transferred:       1731600 bytes
Requests per second:    10.92 [#/sec] (mean)
Time per request:       9154.469 [ms] (mean)
Time per request:       91.545 [ms] (mean, across all concurrent requests)
Transfer rate:          191.05 [Kbytes/sec] received

Memcache is installed, using memcache_storage module, for some reason I got slightly higher requests per second using this module rather than memcache. 4Gb are set as Memcache cache. Here is some Memcache statistics output.

Cache Information
Current Items(total)    10730 (95109)
Hits    644064
Misses  74324
Request Rate (hits, misses) 1.44 cache requests/second
Hit Rate    1.29 cache requests/second
Miss Rate   0.15 cache requests/second
Set Rate    0.19 cache requests/second

APC is also installed, working perfectly.

I don't have many modules installed:

CORE

block, field, field sql storage, field UI, file, filter, image, list, locale, menu, node, number, options, path, system, taxonomy, text, user,

CONTRIB

content access, varnish, ctools, eck, feeds, feeds ui, feeds tamper, feeds tamper ui, email, field permissions, hsm field, node reference, references, user reference, jquery update, mongodb, mongodb storage, mongodb watchdog, entity api, disable messages, image url formatter, job scheduler, libraries, quicktabs, quicktabs styles, apc, eck entitycache, entitycache, memcache storage, services, rest server, entityfieldquery views, views, views json, and views ui.

As for the Devel output, usually I get something like this. (This output was after I refresh the page)

Executed 57 queries in 10.79 ms. Queries exceeding 5 ms are highlighted. Page execution time was 167.27 ms. Memory used at: devel_boot()=1.36 MB, devel_shutdown()=13.96 MB, PHP peak=14.25 MB.

_drupal_session_write query is always the longest, it never executes in less than 8 ms, and sometimes it reaches 500 ms like in the output below.

Executed 57 queries in 607.49 ms. Queries exceeding 5 ms are highlighted. Page execution time was 2395.87 ms. Memory used at: devel_boot()=1.36 MB, devel_shutdown()=13.96 MB, PHP peak=14.25 MB.

These are my PHP settings in settings.php

/**
 *MongDB configuration
 */
 $conf['mongodb_connections'] = array(
   // Connection name/alias
   'default' => array(
     // Omit USER:PASS@ if Mongo isn't configured to use authentication.
    'host' => 'mongodb://localhost',
     // Database name
     'db' => 'mydb',
   ),
 );
 $conf['mongodb_watchdog'] = 'mydb_logs';
 $conf['field_storage_default'] = 'mongodb_field_storage';

/**
 *Memcached
 */
$conf['cache_backends'][] = 'sites/all/modules/memcache_storage/memcache_storage.inc';
$conf['cache_default_class'] = 'MemcacheStorage';
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
$conf['session_inc'] = 'sites/all/modules/memcache_storage/includes/session.inc';
$conf['lock_inc'] = 'sites/all/modules/memcache_storage/includes/lock.inc';

/**
 *Varnish
 */
$conf['cache_backends'][] = 'sites/all/modules/varnish/varnish.cache.inc';
$conf['cache_class_cache_page'] = 'VarnishCache';

/**
 * Add APC Caching.
 */
$conf['cache_backends'][] = 'sites/all/modules/apc/drupal_apc_cache.inc';
$conf['cache_class_cache'] = 'DrupalAPCCache';
$conf['cache_class_cache_bootstrap'] = 'DrupalAPCCache';

Configurations in my.cnf file.

key_buffer_size = 12M

query_cache_size = 32M

query_cache_limit = 2M

query_cache_type = 1

# InnoDB caches.
innodb_buffer_pool_size = 500M
innodb_flush_method = O_DIRECT

max_connections = 75

table_cache = 96

sort_buffer_size = 12M

tmp_table_size = 12M

I really feel lost at this moment, I have read numerous articles about performance and scalability and I feel like I've done most of the things that are mentioned, but don't seem to work at all. I'm stressed because I feel like the server will crash really fast with this numbers. I hope someone can give me advices.

Kind regards.

Answer 1

I think you didn't have any answer yet because your question is way too specific. I would advise you break it down into several precise questions so that you can get...well, precise answers. There's no way someone can provide a perfect answer here.

But let's try and start helping you a bit.

• Since your application will be primarily used by authenticated users and this is where you've noticed performance troubles, I'd suggest you don't focus on Varnish at all for the time being.
• Benchmarking with Apache AB hasn't been the most reliable load testing tool in my experience. I would recommend Apache jMeter instead, and, if you can afford it, try and get a world-class Cloud load testing provider such as Soasta, or, for a less expensive option, try Blitz.io.
• MongoDB: you should try and search for answers on StackOverflow directly where you can get ahold of experienced MongoDB folks there and ask new questions if needed.
  • Note that you'll not find much experience with MongoDB in the Drupal community. I know that there's at least one major site using it for fields storage (examiner.com) but this also requires a pretty deep expertise and is prone to giving you troubles maintaining it in the long run. I'd thus question this whole decision and would step back a little to determine if going with MySQL instead would truly be a bottleneck or if it would simply be slower, but acceptable, still. Smaller batch imports are usually the solution to most Feeds performance issues.
  • Note also that @chx is pretty heavily involved with MongoDB development for Drupal. You'll find 20+ "Drupal 8 progress from my / MongoDB perspective" blog posts on his site.
  • Pretty sure you know about the https://drupal.org/project/mongodb module, but just in case, here you go :-)
• Memcache Storage isn't necessarily the best option. There's been a pretty heated discussion at https://drupal.org/node/2060753 where you'll be able to see that it might be better to fix the main Memcache project rather to try and go with a competing module that is highly unlikely to get the same level of community work than memcache. Also, performance benefits have been widely discussed as the benchmarking methodology used is debatable.

My main advice to you is: keep things simple, remove as many layers of complexity as you can, and start from there. Use the https://drupal.org/project/memory_profiler module to know what pages are very slow, or go to the next level with XHprof profiling (there's a module for that!)