Panda read_csv e UTF-16

Question

Ho un file di testo CSV codificato in UTF-16 (in modo da conservare i caratteri Unicode quando altri usano Excel) ma quando faccio un read_csv con Pandas 0.9.0, ottengo questo errore criptico:

df = pd.read_csv('data.txt',encoding='utf-16',sep='\t',header=0) 
df.head() 

--------------------------------------------------------------------------- 
Exception         Traceback (most recent call last) 
<ipython-input-18-85da1383cd9e> in <module>() 
----> 1 df = pd.read_csv('candidates-spanish.txt',encoding='utf-16',sep='\t',header=0) 
    2 df.head() 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.pyc in read_csv(filepath_or_buffer, sep, dialect, header, index_col, names, skiprows, na_values, keep_default_na, thousands, comment, parse_dates, keep_date_col, dayfirst, date_parser, nrows, iterator, chunksize, skip_footer, converters, verbose, delimiter, encoding, squeeze, **kwds) 
248   kdict['delimiter'] = sep 
249 
--> 250  return _read(TextParser, filepath_or_buffer, kdict) 
251 
252 @Appender(_read_table_doc) 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.pyc in _read(cls, filepath_or_buffer, kwds) 
198   return parser 
199 
--> 200  return parser.get_chunk() 
201 
202 @Appender(_read_csv_doc) 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.pyc in get_chunk(self, rows) 
853   elif not self._has_complex_date_col: 
854    index = self._get_simple_index(alldata, columns) 
--> 855    index = self._agg_index(index) 
856 
857   elif self._has_complex_date_col: 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/io/parsers.pyc in _agg_index(self, index, try_parse_dates) 
980     arr, _ = _convert_types(arr, col_na_values) 
981     arrays.append(arr) 
--> 982    index = MultiIndex.from_arrays(arrays, names=self.index_name) 
983   return index 
984 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/index.pyc in from_arrays(cls, arrays, sortorder, names) 
1570 
1571   return MultiIndex(levels=levels, labels=labels, 
-> 1572       sortorder=sortorder, names=names) 
1573 
1574  @classmethod 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/index.pyc in __new__(cls, levels, labels, sortorder, names) 
1254   assert(len(levels) == len(labels)) 
1255   if len(levels) == 0: 
-> 1256    raise Exception('Must pass non-zero number of levels/labels') 
1257 
1258   if len(levels) == 1: 

Exception: Must pass non-zero number of levels/labels 

lettura dei dati in linea per linea con csv.reader sulla base di this example implica che i miei dati non è correttamente formattato:

from io import BytesIO 
import csv 

with open('data.txt','rb') as f: 
    r = f.read().decode('utf-16').encode('utf-8') 
    for l in csv.reader(BytesIO(r),delimiter='\t'): 
     print l 

['Country', 'State/City', 'Title', 'Date', 'Catalogue', 'Wikipedia Election Page', 'Wikipedia Individual Page', 'Electoral Institution in Country', 'Twitter', 'CANDIDATE NAME 1', 'CANDIDATE NAME 2'] 
['Venezuela', 'N/A', 'President', '10/7/12', 'Hugo Rafael Chavez Frias', 'Hugo Ch\xc3\xa1vez', 'Hugo Ch\xc3\xa1vez', 'Hugo Chavez', 'Hugo Ch\xc3\xa1vez Fr\xc3\xadas', 'Hugo Chavez', 'Hugo Ch\xc3\xa1vez'] 
['Venezuela', 'N/A', 'President', '10/7/12', 'Henrique Capriles Radonski', 'Henrique Capriles Radonski', 'Henrique Capriles Radonski', 'Henrique Capriles Radonski', 'Henrique Capriles R.', 'Henrique Capriles', ''] 

c'è qualche pre-elaborazione, opzione aggiunta a read_csv , o qualcos'altro che deve essere fatto prima che pandas.read_csv possa leggere un file utf-16? Grazie!

Answer 1

Questo è un bug, credo perché il lettore csv stava passando di nuovo una linea aggiuntiva vuota all'inizio. Ha funzionato per me su Python 2.7.3 e 0.9.1 panda se faccio:

In [36]: pd.read_csv(BytesIO(fh.read().decode('UTF-16').encode('UTF-8')), sep='\t', header=0) 
Out[36]: 
<class 'pandas.core.frame.DataFrame'> 
Int64Index: 50 entries, 0 to 49 
Data columns: 
Country        43 non-null values 
State/City       43 non-null values 
Title        43 non-null values 
Date        43 non-null values 
Catalogue       43 non-null values 
Wikipedia Election Page    43 non-null values 
Wikipedia Individual Page   43 non-null values 
Electoral Institution in Country 43 non-null values 
Twitter        43 non-null values 
CANDIDATE NAME 1     43 non-null values 
CANDIDATE NAME 2     16 non-null values 
dtypes: object(11) 

Ho segnalato il bug qui: https://github.com/pydata/pandas/issues/2418 su GitHub dominarlo provoca purtroppo un segfault nel c-parser. Lo aggiusteremo.

Ora, è interessante notare che: https://softwareengineering.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful;)

Answer 2

from StringIO import StringIO 
import pandas as pd 

a = ['Venezuela', 'N/A', 'President', '10/7/12', 'Hugo Rafael Chavez Frias', 'Hugo Ch\xc3\xa1vez', 'Hugo Ch\xc3\xa1vez', 'Hugo Chavez', 'Hugo Ch\xc3\xa1vez Fr\xc3\xadas', 'Hugo Chavez', 'Hugo Ch\xc3\xa1vez'] 

pd.read_csv(StringIO('\t'.join(a)), delimiter='\t') 

lavora qui può caricare il capo dei tuoi dati in modo da poter testare

Answer 3

python3:

with open('data.txt',encoding='UTF-16') as f: 
    df = pd.read_csv(f)