Lately, I have been facing some downtime on some of the Drupal sites I have deployed. I would want to approach the problem pro-actively, rather than reacting to those downtimes by restarting services or the server itself. I have read about monitoring tools like nagios, munin, cacti, etc to monitor the health of your server/infrastructure, but I haven't really tried one in production. I deal with Drupal almost 99% of the times, and I would like to find out suggestions for both server level monitoring, and Drupal (application code) level monitoring.
Basically, I would like to be notified when the server load is high, possibly being pointed to the culprit (or the victim), so that I could take informed decisions. I would also like to know what people use for Drupal level monitoring (such as emailing) when there is a PHP error/warning. (What happens when the email server itself is down?)