French accented characters not saving to database

Question

I have created a module that reads in a csv file that has french accented characters in it to create nodes from. The creation of the nodes seems to work fine but when I execute the node_save() command I get the following error:

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. 
Path: /batch?id=55&op=do StatusText: Service unavailable (with message) 
ResponseText: PDOException: in field_sql_storage_field_storage_write() 
(line 494 of /var/www/html/modules/field/modules/field_sql_storage/field_sql_storage.module). 

The special character I am having problems with is é character (however I am sure other accented characters will cause me grief as well).

I thought this was an encoding issue and checked the database encoding on the tables - all tables are set to UTF8. I believe that this is the correct encoding and should work (however I am no database expert).

I tried using the following code:

$encodedval = mb_check_encoding($val, 'UTF-8') ? $val : utf8_encode($val);

Which seems to help but still gives the same error after 300 items are read in (there are 3000 items that need to be imported). During the importing, I am Geo-referencing the nodes, so could this be Google killing the look-ups after so many hits?

I am at a loss and would appreciate if anyone could shed some light as to what might be the issue.

Answer 1

Both suggested solutions above are good things to check when trying to solve this problem.

In my particular case, using the fgetcsv command. It silently chokes on files that are not UTF-8 encoded and contains accented characters. To fix this problem I had to implement the file read function as follows:

while (($line = fgets($file_handle)) !== FALSE) {
  // Ensure the line we read is encoded properly
  $value = mb_check_encoding($line, 'UTF-8') ? $line : utf8_encode($line);
  $data = str_getcsv($value);
  // process the data as normal
}

This fixed my issue.

Answer 2

Have you tried to save your module file in another encoding format?

I'm usually having the same issues as my files are default to AINSI. After converting the file to UTF-8 I have no more issues.

By doing so you will have to remove your utf8_encode() calls to avoid double encoding.

Answer 3

I've imported thousands of Spanish-language nodes using the Feeds module; so I would indeed suspect that something is going wrong in the georeferencing, especially with the "Service unavailable" message. Does Google return something human-readable? If so, I suggest writing out the responses to a log file somewhere, and see if something goes wonky after 300 requests.