I have been updating some modules including Views and Feeds. Given these modules are quite complex, I thought a good method of testing whether the update was successful or not was to run the tests which come as part of the module. This can be performed by using simpletest module which comes with Drupal core.
However the tests do not complete successfully (by this I mean that there was not a 100% pass rate). I then rolled back to the previous version of Feeds, and low and behold it did not fully pass either. This is a large site which has many visitors and I am certain that the Feeds functionality has been running fine, otherwise someone would have been complaining about it.
So it seems Feeds is working on the production site, but the built-in tests don't fully pass.
As follow up I did a clean install of Drupal 7.38 and then added latest versions of Feeds, Views, Features, ctools, Job scheduler and Feeds Tamper. Site is a standard install with no content. After running the suite of feeds tests I get even more failures than on the existing site I mentioned previously. In fact almost all the tests fail.
This has got me wondering if in-built tests are a good measure of whether a contrib module is working correctly? Or are they quite unreliable, as I now suspect?