You should consider Drupal 6 without phpass to be unnecessarily insecure. And yes, you should use phpass on all your D6 apps. There are reasons they've changed the way things work in Drupal 7, and made the same changes available as a module to D6 users.
There is no silver bullet, but there's no reason to ignore an easy-to-use module that will significantly increase your level of security.
Not using it is a little like not wearing a seat belt: of course you want to avoid ever being in an accident to begin with (analogous to ever letting anyone gain access to your data), but once you have an accident it's far too late to try buckling up. Buckle up now!
Update: A security expert seems to agree with me:
...take the view that attackers will ultimately get into your network, access your data... you should be looking at the problem differently.