I would like to load my own, but Drupal seems to forcefully load it's own by default behind the scenes.
Turned out to be as simple as placing favicon.ico into the theme root folder (!)
http://YOURSITE/admin/build/themes/settings - upload your favicon for all enabled themes, as for "Global settings".
In Drupal 7 there's a way that's easier than loading the image file to a particular directory and you get some error checking. There's a global admin page that allows you to upload in icon image. Log in as an administrator and visit
Look near the bottom of the page for the "Shortcut Icon Settings". You can turn on/off the default, set a path to a file and upload images.
Each theme has a similar setting that overrides the global setting. Be aware that changing the global setting my not work as you expect for a changed theme.