There are three things that I would check.

One, if you are on a production site and not editing PHP files, then you should ensure that APC is enabled, has enough memory, and has a long TTL (you could go with a day or never expire if you wanted). The APC docs have all of the information you need for setting the TTL. For choosing the amount of memory, you should stick the apc.php file somewhere protected and monitor the memory usage and churn statistics. Adjust the APC memory so that your miss rate is very low. Initial slowness could be because APC is full and emptying out (IIRC, APC dumps the entire cache when it is full instead of employing LRU or more advanced cache strategies).

Second, make sure you have MySQL tuned appropriately. You can use mysqltuner to adjust your buffer sizes. Your initial slowness could be because of loading tables from disk and/or query cache misses. While not perfect, mysqltuner does get you going in the right direction.

Third, make sure you have a read Drupal cron strategy. Personally, I would disable automagic cron on "admin/config/system/cron" and set up a crontab to run every night. You can also try out Elysia Cron if you really do need finer grained control over things. This way you can the necessary tasks run as often as you need, but have the normal ones run overnight. Your initial slowness could be from cron runs happening every hours. You can confirm this by looking at when cron runs on "admin/reports/dblog" and trying to match up with your slowness.

There are three things that I would check.

One, if you are on a production site and not editing PHP files, then you should ensure that APC is enabled, has enough memory, and has a long TTL (you could go with a day or never expire if you wanted). You can also consider setting apc.stat=0. The APC docs have all of the information you need for setting the TTL. For choosing the amount of memory, you should stick the apc.php file somewhere protected and monitor the memory usage and churn statistics. Adjust the APC memory so that your miss rate is very low. Initial slowness could be because APC is full and emptying out (IIRC, APC dumps the entire cache when it is full instead of employing LRU or more advanced cache strategies).

Second, make sure you have MySQL tuned appropriately. You can use mysqltuner to adjust your buffer sizes. Your initial slowness could be because of loading tables from disk and/or query cache misses. While not perfect, mysqltuner does get you going in the right direction.

Third, make sure you have a real Drupal cron strategy. Personally, I would disable automagic cron on "admin/config/system/cron" and set up a crontab to run every night. You can also try out Elysia Cron if you really do need finer grained control over things. This way you can the necessary tasks run as often as you need, but have the normal ones run overnight. Your initial slowness could be from cron runs happening every hour. You can confirm this by looking at when cron runs on "admin/reports/dblog" and trying to match up with your slowness.