The code used by Drupal to delete temporary files is the following one. (It runs during cron tasks.)
// Remove temporary files that are older than DRUPAL_MAXIMUM_TEMP_FILE_AGE.
// Use separate placeholders for the status to avoid a bug in some versions
// of PHP. See http://drupal.org/node/352956.
$result = db_query('SELECT fid FROM {file_managed} WHERE status <> :permanent AND timestamp < :timestamp', array(
':permanent' => FILE_STATUS_PERMANENT,
':timestamp' => REQUEST_TIME - DRUPAL_MAXIMUM_TEMP_FILE_AGE,
));
foreach ($result as $row) {
if ($file = file_load($row->fid)) {
$references = file_usage_list($file);
if (empty($references)) {
if (!file_delete($file)) {
watchdog('file system', 'Could not delete temporary file "%path" during garbage collection', array('%path' => $file->uri), WATCHDOG_ERROR);
}
}
else {
watchdog('file system', 'Did not delete temporary file "%path" during garbage collection, because it is in use by the following modules: %modules.', array('%path' => $file->uri, '%modules' => implode(', ', array_keys($references))), WATCHDOG_INFO);
}
}
}
file_usage_list() is the function that queries the "file_usage" table.
The function that adds a row in that table is file_usage_add(); looking at the functions/hooks that calls it, I conclude that the "file_usage" table is used for managed files. Since you are not deleting the files with a row in the "file_managed" table, I would say you can avoid checking the "file_usage" table.