2

I have been updating some modules including Views and Feeds. Given these modules are quite complex, I thought a good method of testing whether the update was successful or not was to run the tests which come as part of the module. This can be performed by using simpletest module which comes with Drupal core.

However the tests do not complete successfully (by this I mean that there was not a 100% pass rate). I then rolled back to the previous version of Feeds, and low and behold it did not fully pass either. This is a large site which has many visitors and I am certain that the Feeds functionality has been running fine, otherwise someone would have been complaining about it.

So it seems Feeds is working on the production site, but the built-in tests don't fully pass.

As follow up I did a clean install of Drupal 7.38 and then added latest versions of Feeds, Views, Features, ctools, Job scheduler and Feeds Tamper. Site is a standard install with no content. After running the suite of feeds tests I get even more failures than on the existing site I mentioned previously. In fact almost all the tests fail.

This has got me wondering if in-built tests are a good measure of whether a contrib module is working correctly? Or are they quite unreliable, as I now suspect?

Improve this question
1
  • They was designed for different thing, so it dies not surprise me. – Mołot Aug 7 '15 at 9:29
1

No

Automated tests was created to test code, and to ensure quality of patches. They are meant to be executed in specific, clean environment (e.g. Testbot environment), which your live site certainly is not.

Second reason why it's no is what is tested. These tests only test code. But downloading and unpacking code is simple. Not many thing that can go wrong, and if it goes wrong, you will rather see. The hard and important part is database upgrade - and tests will not try to verify if your data was updated properly.

Improve this answer
2
  • Thanks for the reply. You mention that tests are meant to be executed in specific, clean environment. What would constitute a clean environment? Wouldn't a brand new virtual machine with a virgin install of Drupal be considered a clean environment? I am puzzled that a brand new installation without any users or content (i.e. hasn't been touched after the initial installation was completed) would not be considered a clean environment. – Benjen Aug 10 '15 at 10:19
  • @Benjen By specific I meant testbot environment. AFAIR there are some things, like known preset passwords and roles, your brand new instance don't have. – Mołot Aug 10 '15 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.