Why are we using utf8mb4_general_ci and not utf8mb4_unicode_ci?

Question

Drupal is moving to support utf8mb4, however, it is using utf8nb4_general_ci. Why take the time to move over to support it, and then not fully support it? Is there a specific reason, or just continuing utf8_general_ci from previous?

Would there be any problems with ignoring this and using unicode anyway?

Drupal Ticket: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols)

Here is a question on stack exchange which says really there is no reason not to use unicode now days and finally, on this question, it says "utf8_general_ci is a legacy collation that does not support expansions, contractions, or ignorable characters."

I note that WordPress uses utf8mb4_unicode_ci.

Answer 1

It seems to me that the recommendation is outdated and that utf8mb4_unicode_ci will work without problems. It has been used by a lot of people for a long time.

---

There is a difference between changing the character set from utf8 to utf8mb4 (to support more codepoints) and changing the collation from general_ci to unicode_ci (to get more accurate sorting). Both changes can cause their own problems, so doing both independently makes sense.

utf8mb4 is used by default since 8.0.0-beta12. The main issue seemed to be a change of key lengths limitations for InnoDB but as I understand it, utf8mb4 should have worked with the default MyISAM engine even before that change.

Switching to unicode_ci shouldn't cause problems, but may unexpectedly changes the order of sorting for some sites.

The default collation setting is just a default and modules can choose their own collations anyway if they need to. I also haven't found any documentation that says modules should expect a certain collation. The database install guide just lacks a clear statement about which collations are supported and is inconsistent:

• In the section about phpMyAdmin it says that you have to

  Make sure you select COLLATION utf8_general_ci

• Later in the section about installation from command line, general_ci doesn't seem to be required and any UTF-8 collation will do:

  Note: The database should be created with UTF-8 (Unicode) encoding, for example utf8_general_ci.

Furthermore, PostgreSQL is supported and it seems its default UTF-8 collation is equivalent to utf8mb4_unicode_ci, so using that with MySQL should be fine too.

Answer 2

Below link explains that utf8mb4_unicode_ci is better than utf8mb4general_ci (which is a little bit faster) because the second one have problems in sorting order in some languages: https://stackoverflow.com/a/766996/860099