How can I remove emoji from all user input?

Question

It looks like iPhone emoji are not currently supported in Drupal 7 on MySQL.

When I try to save a node containing an emoji (I 💛 U!) in the database, I get an SQL error:

PDOException: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\x9F\x92\x9BRi...' for column 'body_value' at row ...

So, to avoid this very unpleasant user-unfriendly error, I would like to automatically clean all user input, including all node textfields and private message module messages.

I found several functions for removing emoji, but which hook can I use to clean all user input?

Answer 1

All user input is a bit much. Only common ground I can see is that all user input goes thorough some kind of forms. And the only thing that for sure happens before anything tries to save, is validation*.

Functions listed in #validate can alter form state. Example definition shows they take form state by reference:

function test_form_validate($form, &$form_state) {

So the "only" thing you would have to do is to recursively iterate over all $form_state['values'], and apply your removal function. It should be pretty safe, because by the very nature your function will not affect numerical inputs in any way.

Other approach might be a bit faster. You can use #element_validate - prepend your removal function to validation array of each and every text field. Sadly, form element types may be added by modules, so you would have to prepend it to every element anyway (forgetting any significant boost due to validation only data that needs it, and paying a price of more function calls), or carefully inspect your modules and make sure you know what else, except textfield and textarea can accept text data (machine_name element is already safe, I omitted it on purpose).

All in all, I would prefer to just switch my MySQL to utf8mb4 charset, that supports all utf8 as it ought to be supported. You need MySQL 5.5.3 or greater to do this.

• make a backup (obvious, but hey)
• upgrade if needed
• for each database, table and column that has it set already, change to CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci
• Set default-character-set = utf8mb4 for [client] and [mysql] in my.cnf

• Set (in the same file)

  [mysqld]
  character-set-server = utf8mb4
  collation-server = utf8mb4_unicode_ci
  

• Hack Drupal to use it. What you want is around line 53 of /includes/database/mysql/database.inc

Hacking Drupal is usually not encouraged, but in this particular case we cannot expect a patch, so this is the only way to give users what they want. Also, it's method way less CPU consuming that altering all inputs.

---

* Of course you may create your own submit handler and make it a first one, but making sure it's always first is hard. Other module developers may try to have their handlers first, too. And if they decide to, for example, log something to MySQL, your fix becomes futile. Using validation is a bit of abuse, but it's an only way I would trust in this scenario.

Answer 2

You could try the Emoji Scrub module.

Answer 3

If you want to remove/replace emoji with some other text you can use the Strip 4-byte UTF8 module.

This module helps in preventing PDO Exceptions caused by MySQL general error of incorrect string value. Enabling this module will have your site reject overly long 2 byte sequences, as well as characters above U+10000, and reject overly long 3 byte sequences and UTF-16.

I've used the Unicode module for one of my project.It deals well with emojis

Unicode module converts unicode characters to html entities so that drupal may store them correctly in mysql.

This module only does this conversion prior to saving entities by looping over the defined properties of the entity and only converts properties of type text & formatted text.