Let's imagine a site which is almost completely static. There are news which are rarely updated and from time to time some new content in the form of more news. If there are only anonymous users apart from the admin... would the cache retrieval be so fast as simple HTML pages? I ask this because I have tested the config above with cache on and it goes very slow in comparison with HTML saved counterparts and I am not sure if it is a misconfiguration.
No it would be unlikely to be as fast as plain HTML, because the default caching is still using the database and some PHP.
However you can improve the speed significantly with a module or two. It is highly dependent on the speed of your servers - if you have a fast database server but a slow filesystem you might be better using the database although that is an unlikely situation. At minimum I recommend installing Advagg - correctly configured that will reduce load times for a lot of sites by around 25%.
Depending on the site, Boost can also vastly improve performance although I have not found it as universally applicable as Advagg. Also depending on your setup an external cache like Varnish may be better although it requires careful configuration and some server admin knowhow.
Your problem might be misconfiguration of some modules, perhaps overuse of others. If you are using a shared host the problem might lie on the PHP Memory Limit, that sometimes is quite low (although lots of servers nowadays will allow you to change this for free).
You need to find out where is your bottleneck. Might be the connection with the db, the speed and limits of the PHP, perhaps some custom module...