6

How can I tell SimpleTest to use a different database than the default connection? I want to be able to run the tests against an in memory database while keeping the local data on disk.

UPDATE:

So far the answers I'm getting are on how to set up tmpfs and move mysql data to that folder. That's great and super helpful. However this is missing the specific answer that I am looking for. I'm most interested in how to run SimpleTest against a different connection than the default database connection. You can see above that my question reflected by both the title and the first line of my original submission. I apologize for the confusion caused by my mention of an in memory database, I provided that only as context, but it is not the root of my question.

  • What do you mean by "in memory database" ? RAM database ? – B2F Jul 9 '12 at 18:22
  • @ZenMaster dev.mysql.com/doc/refman/5.0/en/memory-storage-engine.html Yes OP meant RAM db. @ greggory How does it matter if its in disk or memory as simple test tables will be cleaned up – GoodSp33d Jul 9 '12 at 18:28
  • yes, this is ture, but it takes simpletest a good 2 and a half minutes to set up with all the CREATE calls. Writing to RAM should be quite a bit faster than writing to disk. It's 100% an issue of test performance. – gregghz Jul 9 '12 at 18:30
  • Copied the description that I already had in a comment of my answer to the actual answer about how to change the connection. It's not in any complicated, you just need to copy a lot of code around and then adjust it. – Berdir Jul 11 '12 at 19:00
4

It's not possible to do this on a storage engine level. You can't install Drupal on top of the HEAP storage engine, there are various missing features, the most obvious one being: "MEMORY tables cannot contain BLOB or TEXT columns." (copied from the link in the comments).

The qa.drupal.org testbot's have the complete MySQL data directory in a tempfs that is stored in memory, which is copied from the real file system during startup and back during shutdown. I don't think that's a doable setup for a development environment because, at least in my case, databases of real sites quickly get too big for something like this (Working on large sites with multiple GB of data). http://drupal.org/node/466972 has some more tipps and scripts for the tempfs thing. Make sure to read the comments.

Not sure what exactly takes 2,5m in your case, I can run a single test in ~10s (Note that every test method is a separate test that runs in a new installation).

I'm quite sure that I've already written quite a list of tipps on how to speed up tests in an answer somewhere but I'm unable to find it. The most important thing is defining $profile = 'testing'; in your test class, that will only install the absolutely required modules for Drupal to run. Speeds up test runs by ~50%. Another way is prefixing your different test methods with anything other than test and having a single public test method that calls all other. Downside is that your tests then run in the same environment and might affect each other. You e.g. can't have hardcoded node id's and things like that.

Another thing is optimizing your mysql configuration, than can speed up tests quite a bit as well, see http://techblog.md-systems.ch/blog/improve-mysql-performance for a start.

To actually override the used database connection, you need to completely overwrite the setUp() method where the connection is created and activated and instead use whatever connection you want, see DrupalWebTestCase::setUp().

  • I wasn't aware that each test method rebuilt the test environment (still, the long times I'm getting have been with only one test method). Can you elaborate on $profile? Is that just a member of the test class? Is that not already set in DrupalWebTestCase for you? – gregghz Jul 9 '12 at 19:34
  • Also, I think tempfs is what I had in mind and not the HEAP storage engine. But more than just moving everything to RAM, I'm curious how I can have one drupal install use a different database connection when running tests. That's my main question, the bit about in memory database stuff is just for context. – gregghz Jul 9 '12 at 19:36
  • Yes, it's a class property. In Drupal 7, $profile is set to standard by default. That means each test will enable all modules defined in the standard profile (21 modules), which is a ton more than the testing profile which has exactly zero additional required modules. Note that this even excludes basic modules like block, if you have e.g. tests for your custom blocks, you will need to explicitly enable it. – Berdir Jul 9 '12 at 19:41
  • When using InnoDB, all data is stored in the same file, it doesn't matter if it's a different database or not. The database-name directory is only used for MyISAM. So you'd need to "hack" core to default to MyISAM when running tests (I'm actually doing this, because MyISAM creates tables faster). Then you'd need to override api.drupal.org/api/drupal/… and instead of copying the default connection, create one from scratch or reference an existing from your settings file. – Berdir Jul 9 '12 at 19:45
  • 1
    The link in my answer contains scripts for the tempfs thing, but I haven't tried them myself. And as said, they move the complete mysql data directory into tmpfs, which might be huge. As a start, try to optimize your mysql configuration first, see techblog.md-systems.ch/blog/improve-mysql-performance. – Berdir Jul 9 '12 at 20:22
3

I am now running SimpleTests with MySQL on a tmpfs folder, and the execution time went from 27 seconds to 5 seconds. I'll explain how I did it.

  • Create a tmpfs folder

    mkdir /var/tmpfs
    mount -t tmpfs -o size=1G tmpfs /var/tmpfs
    chown -R mysql:mysql /var/tmpfs/mysql
    
  • Copy your MySQL data into the tmpfs directory

    sudo service mysqld stop
    cp -Rv /var/lib/mysql /var/tmpfs/
    chown -R mysql:mysql /var/tmpfs/mysql
    
  • When you want to run some tests, configure my.cnf to use tmpfs.

    #/etc/my.cnf
    [mysqld]
    datadir=/var/tmpfs/mysql
    
  • Start mysqld

    sudo service mysqld start
    

The data won't be persistent but it shouldn't be a problem since the website used for testing should not be in a production environment. Anyway you can always switch back to datadir=/var/lib/mysql when you need to save data.

Note that it still worth adding the 'testing' profile to the DrupalTestCase class

protected $profile = 'testing'; 

Indeed, on the same without the above it went up to 12 seconds, instead of 5 seconds.

  • When attempting to start mysql after following these instructions I get "start: Job failed to start" – gregghz Jul 10 '12 at 20:19
  • from this ubuntuforums.org/showthread.php?t=1469902 I think you typed start mysql where I really meant "sudo service mysqld start", same for stop, or restart – B2F Jul 10 '12 at 20:34
  • I'll try again, but I don't think that was the issue. I'm fairly certain I wasn't using the upstart version and it started fine when I set datadir back to the default. – gregghz Jul 11 '12 at 14:45
  • I just tried it again and it doesn't properly start when I point datadir in my.cnf to /var/tmpfs/mysql I get "start: Job failed to start" after running "sudo service mysql start" (mysqld is not what is installed for me). But it starts fine when not using tmpfs. Also, in the bit you have to copy the data, you have cp -Rv /var/lib/mysql /var/ramfs ... shouldn't that last bit be /var/tmpfs? – gregghz Jul 11 '12 at 16:18
  • You are absolutely right, it's tmpfs, I edited out the ramfs typo. It still doesn't work? Maybe fstab missing but I don't know – B2F Jul 11 '12 at 17:47
2

If you want to perform tests on the current database (usefull to test your specific cases of data migration) you could override DrupalWebTestCase::setUp() and also DrupalWebTestCase::tearDown(). Here some exemple by Trellon.com http://www.trellon.com/content/blog/forcing-simpletest-use-live-database

As for me I use something like this:

abstract class DwCurrentDbBasedTestCase extends DrupalWebTestCase {

  /**
   * Override setUp().
   *
   * Avoid creating a new database and install for testing.
   * We want to use current. Our CI will check project build.
   */
  protected function setUp() {
    global $conf;

    $conf['error_level'] = 2;

    $this->originalFileDirectory = variable_get('file_public_path', 'sites/default/files');
    $this->public_files_directory = $this->originalFileDirectory . '/simpletestFiles';

    // Use the test mail class instead of the default mail handler class.
    variable_set('mail_system', array('default-system' => 'TestingMailSystem'));

    // Simpletest needs this to run any tests
    $this->setup = TRUE;

    // @see drupal_web_test_case.php
    $modules = func_get_args();
    if (isset($modules[0]) && is_array($modules[0])) {
      $modules = $modules[0];
    }
    if ($modules) {
      $new_modules = array();
      foreach ($modules as $module) {
        if (!module_exists($module)) {
          $new_modules[] = $module;
        }
      }
      if (count($new_modules) > 0) {
        $success = module_enable($modules, TRUE);
        $this->assertTrue($success, t('Enabled modules: %modules', array('%modules' => implode(', ', $modules))));
      }
    }
  }

  /**
   * By default this method will remove testing env, avoid this in below code.
   */
  function tearDown() {
    // In case a fatal error occurred that was not in the test process read the
    // log to pick up any fatal errors.
    simpletest_log_read($this->testId, $this->databasePrefix, get_class($this), TRUE);
  } 
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.