Import issue in other languages using csv file to drupal node creations

Question

I am trying to import a csv file using parsecsv-for-php library.The given csv file is in Catalan language.

The thing is that when i just try to create nodes with given csv data it is throwing me this error like this while importing PDOException: in field_sql_storage_field_storage_write() (line 448 of /modules/field/modules/field_sql_storage/field_sql_storage.module).

To resolve this issue, what i did is that used utf8_encode function for each fields and avoided that error.

After successfully importing, the issue over here is that, this is the body text in csv

Thomas Alva Edison va anunciar la invenció del fonògraf l’any 1877, fent una demostració aquest mateix any. L’any 1878 el va patentar als Estats Units. Inicialment per enregistrar va fer servir cilindres de cartró recoberts d’estany, per finalment utilitzar cilindres de cera sòlida. Aquests van ser comercialitzats l’any 1889. La competència amb els discos sonors va acabar per eliminar els cilindres de cera cap els anys 1920s. Tot i així van sobreviure uns anys més com a suports d’enregistrament a oficines amb el dictàfon.

and the imported body text in the drupal i have displayed as image

If u compare both u will know there are certain differences in the body text such as

l`any in csv body text is getting imported in drupal as lany

d`estany is getting imported in drupal as destany

d`enregistrament is getting imported in drupal as denregistrament

I just want the same text to imported in drupal too with out missing any symbols or special characters. what am i doing wrong.

I have attached the my uploaded csv file formats too.

Only this character(') is not coming up properly in drupal.Any help on this is really appreciated.Thanks in advance

Answer 1

I found a way to overcome this issue. What I exactly did is that, in the imported view, I just added another charset=ISO-8859-1 which retrieves back the symbols or characters from db and displayed them properly.

<?php
drupal_add_http_header('Content-Type', 'text/html; charset=ISO-8859-1,utf-8');
$new_output = utf8_decode($output);
echo $new_output;
?>

That put back all special characters or symbols which was missing.

Answer 2

Can you check what markup gets saved to database? The table field_data_body seems to be in utf8 by default. That would rule down the possibility of field_formatters messing the characters in presentation layer.

If the data in database is corrupted, try getting the output of utf8_encode logged to some file for example. Is it possible to encode the csv to utf8 with some external tool before using the import tool?

File encodings are tricky and even using wrong encoding in module file can alter the markup. I need to output some xml-files as ISO-8859-1 to serve another system updated content from my service and even using ide-with default encoding can break the modules output by chanching file format. So narrow down more and you will find the break point.

The original pdo-exception could be indication that ´-char was read correctly, but assumed to be text separator. Using escaping should work in that case. In that case run the original importion with using something like $row_field = '"'.addslashes ( $row_field ).'"'; before saving to database.

Alternative to addslashes would be check_plain($row_field) .

If these dont work then its quite possible that the database doesnt support 'utf8mb4, which is "real" '.