Here are a couple of the more obvious tips:
Enable caching for your Drupal.
Optimize your Drupal database.
If your Drupal is hosted on a dedicated solution, you can modify the global MySQL settings for better performance.
*More for front-side micro-optimisation, you can check your website using this online tool. It will give you valuable information on what modifications can be made in order to improve its performance.
Of course, disable/remove all modules/blocks you are not using.
Now, most of those are pretty basic and usual optimisations. I assume this is where you are now standing.
In my (humble) opinion, you would get way better response times if you install APC or Memcache on your server. There is also varnish I have heard lots of good things about, but I have not had the chance yet to play with it, and it looks like it is meant for anonymous sessions and static pages so I am not certain you would gain as much benefits than from other 2 above mentionned modules.
If you want to see benchmarks of a Drupal instance with/without APC caching, see here.
Last but not least, have you tried benchmarking without the flag module to see what impact this module has? We had to rewrite some queries of the like in other 3rd party modules a few times, some of the sql Drupal generates is not really optimised as you probably realised by now.
Happy coding, cheers!