EDIT: I misread the original post. 168 modules is a lot, and 300 to 700ms of SQL queries is huge. The more modules you will use, the more queries they will be as soon as modules does some.
Use aggressive caching while you can, cache everything, if it's not enough, try a reverse proxy cache. Using a CDN for files can greatly improve the whole thing. A reverse proxy cache can also help you by removing some auth cookies when hitting pages that does not needs it (then core will think user is anonymous for those and maximise caching).
The Drupal core dynamism make the whole dawn slow as soon as you have too many modules interacting at the same time.
I'd say, for example, if you use a lot of modules that loads data at hook_node_load() time instead of using fields, it will make a lot of queries while the field usage would have ensure caching efficiency.
Rendering can take a lot of time also, drupal_render() (the rendering API sometime being called) is a nice piece of API (really useful) but also a bit slow. Switching to PDO (D7) and the full DBTNG (which is great by the way) also add non neglictable latency.
That said, core by itself is quite fast (but it does too much SQL queries, even with almost nothing installed), poorly coded modules often are the bottleneck.
APC can divide execution time per 2 or 3, depending on the code that runs. if you configure it well (enable all APC optimizations, the offical APC manual is well written and will guide you).
If you are on a box with a slow file system (network file system or slow hard drive) it can imply a visible impact on execution time. Drupal is made from a lot of small files, which forces PHP to do I/O on the FS each time it loads one of them (APC also helps a lot for that).
A misconfigured DBMS can also be quite an ugly bottleneck, if you are using MySQL think about doing fine tuning. If you are on a shared hosting, if it's not Drupal specific (or ready) DBMS and PHP stack will probably be misconfigured or non-tuned, which can lead to really slow sites.
Don't forget to activate all the caches. If your site is not authenticated user oriented, then activate the aggresive page caching (it really is amazing).
The more you'll have blocks, the more full pages will be slow, Views' module blocks will be a dawn bottleneck (depending on the Views plugins you use, OG's block can be a real pain) if you don't restrict their visibility on a per-page basis, or with custom PHP code (any other block also, always set your block visibility manually, greatly helps the framework by avoiding it to attempt to render empty blocks).
Avoid modules that uses hook_init(), hook_init() is being run on every page, even if you get a 403 or a 404, which slows down everything (it even slow down imagecache|style generation time, and 404 errors on files would be dawn slow just to tell you the file doesn't exist).
