I have some JS and CSS on my site that are very specific to some pages. From a Drupal perspective, it is good idea to add them using
drupal_add_js() instead of
THEME.info? Does such thing affect caching, or anything else Drupal-specific?
This question is borderline one, it's one of these topics where art meets science of software development.
For site-wide vs single page: you probably have some kind of JS and CSS aggregation enabled. Using different sets of files is wasteful - it means that for every page your client needs to download another aggregated file. Leaving the same set everywhere allows clients to use cached aggregated version - and this is (usually) the least time-consuming and least transfer-consuming way to go.
As Clive reminded, there are situations that are exceptional. Sometimes you need different versions of jQuery, or conflicting sets of libraries, and aggregating them can result in JS error, or create really big aggregated file. In situations like this you may not want to include all of them, or simply it might be impossible due to conflicts. Even then minimizing the number of "sets" is a good idea.
If you have a really big script used only on one rarely used subpage, some aggregation modules will allow you to exclude it from aggregation and always serve separately - a wise choice if you have 1kb of JS logic that you need in one place only, for example. This part is an art of profiling, not really knowledge and recipes.
If you aren't, then the question isn't really Drupal specific. Each JS file will potentially result in a HTTP request, depending on cache headers and what your browser does. In general, you want to keep the number of requests low, and also try to transfer as few bytes as possible.
If you are using aggregation, then things get tricky. If you use per-page files, then you will have different aggregated files, which means that your users will end up making HTTP requests for them. In other words, this can defeat the effectiveness, somewhat, of file aggregation.
As with all things, proper profile on your site is the only way to tell whether it will make a difference for you.