How do I solve this problem?
You can guess all you want and experiment, but the only way to actually achieve meaningful results is to profile the system to figure out where the bottlenecks are. There are many ways to accomlish this.
You can use the Performance module to identify slow pages.
Are you running APC? If not, you should. Stick the apc.php file somewhere safe and check the statistics to make sure you don't have cache churn.
You can also read up on profiling and optimizing Apache
You could also experiment with Zend Server CE as it includes a PHP optimizer in addition to opcode caching.
In addition to what MPD writes: before you have a look at tools that complement Drupal, have a look at the settings in Drupal. Make sure you enable caching and CSS/Js aggregation on /admin/config/development/performance.
Also check your theme settings: some themes (like Zen) have a developer mode which forces the theme registry to be rebuilt on every page load, which can be handy for developing, but a huge performance hit at the same time.
Minify module also minify HTML.