Out of the box, Drupal supports MySQL (or an equivalent such as MariaDB), PostgreSQL and SQLite for its main database. With additional drivers it also supports Microsoft SQL server and Oracle Database.
The MongoDB module can be used to off-load storage of things like cached data, fields (ie. most data for content, user, taxonomy terms, etc.) and sessions data from the main database to MongoDB. But MongoDB alone can not be used, you still need a SQL database. See NoSQL vs Other SQL Drupal Setups and its accepted answer for more information.
I'm not an expert, but except for SQLite, I'm not sure switching DB engine is a good way to get great performance gain. I guess each engine has its strengths and its own optimization tools. I guess great performance gains can be obtained in each of them, but you have to knows how to do it.
There are also others performance aspects to consider, such as caching layers (Varnish, Memcache, APC, etc.), clustering and Content Delivery Network for static files.