When you uninstall a module, you need to first disable it from admin/modules and then uninstall it from admin/modules/uninstall. The same goes for themes, which should be first disabled from admin/appearance.
If you, for example, delete the module without disabling the module, Drupal will continue to search for the module in the file system, which can result, among other errors, in the issue you are experiencing.
See also Avoid re-scanning module directory when multiple modules are missing.
To understand which module has been deleted without first uninstalling it, you will need to run the SELECT name, filename FROM system WHERE status = 1;
query and compare that list of module with the modules present on the site directory. All the enabled modules and themes should be present.
There's also a hook implementation Martin_Dresden posted on Drupal 7 abysmally slow: file_scan_directory called > 1000 times per page load that may help with debugging. I provide here a slightly modified version.
function MYMODULE_init() {
$startingtime = microtime(true);
$o = '<p>Checking for dead modules ...</p>';
$result = db_select('system')
->fields('system', array('filename'))
->condition('status', '1', '=')
->execute();
$n = 1;
$m = 0;
foreach ($result as $row) {
$path = DRUPAL_ROOT . '/' . $row->filename;
if (!file_exists($path)) {
$o .= "#$n $path<br>";
$m++;
}
$n++;
}
$timedif = round(microtime(true) - $startingtime,3);
$o .= "Total of $n active modules registered in database. $m dead entries found.<br>";
$o .= 'Query Time: ' . $timedif . ' seconds';
drupal_set_message($o);
}