I'd probably do this in the theme with template_preprocess_panels_pane() - http://drupalcontrib.org/api/drupal/contributions!panels!panels.module/function/template_preprocess_panels_pane/7
You can do (make sure you have devel.module installed first)
and get a full print out of everything that's going on in all your panel panes. With enough digging, you should be able to find a
$vars['_something_']['#attached']['js'] = YOUR_FILE; to hook into. I think you can do this in a module too, if that's what you want.
EDIT: After actually thinking about it though..
I would probably advise against this and just put it in your theme js. Unless it's an enormous amount of js, you're going to be taking better advantage of Drupal's js aggregation by not doing this. If you put the js in your theme, it'll be compiled into the main site js file automatically, whereas if you do something different when only this panel pane is showing, it'll require that an entirely different js file be aggregated and downloaded on pages when it shows.
Assuming this is a performance tweek, I think you might actually be headed in the wrong direction with this one, since browsers should cache this file after the first time it's downloaded.
edit: hook name typo