problems with CSV module
in [Python]
|
From: Carlos Grohmann on 3 Jun 2010 14:55 Hi all, I'm using csv to read text files, and its working fine, except in two cases: - when there is only one line of text (data) in the file - when there is a blank line after the last data line this is the kind of data: 45 67 89 23 45 06 12 34 67 .... and this is the function: def getData(paths): """get data from file and create lists with values and column names.""" filehandle = paths#[i] # filehandle = os.path.join(dirname, filename) csvfile = open(filehandle,'r') # Open the file and read the contents sample = csvfile.read( 1024 )# Grab a sample csvfile.seek( 0 ) dialect = csv.Sniffer().sniff(sample) # Check for file format with sniffer. csvfile = csv.reader( csvfile, dialect ) if csv.Sniffer().has_header( sample ): #if there is a header colnames = csvfile.next() # label columns from first line datalist = list( csvfile ) # append data to a list else: #if there is NO header datalist = list( csvfile ) # append data to a list colnames = ['col_%i' % i for i in range(len(datalist[0]))] # label columns as col_1, col_2, etc return datalist, colnames TIA for any help.
From: Neil Cerutti on 3 Jun 2010 15:14 On 2010-06-03, Carlos Grohmann <carlos.grohmann(a)gmail.com> wrote: > Hi all, I'm using csv to read text files, and its working fine, except > in two cases: > > - when there is only one line of text (data) in the file > - when there is a blank line after the last data line > > this is the kind of data: > > 45 67 89 > 23 45 06 > 12 34 67 > ... That data doesn't appear to be csv worthy. Why not use str.split or str.partition? > and this is the function: > > def getData(paths): > """get data from file and create lists with values and column > names.""" > > filehandle = paths#[i] # filehandle = os.path.join(dirname, > filename) > csvfile = open(filehandle,'r') # Open the file and read the In Python 2.6 and earlier, you need to open the file in binary mode. In Python 3.0 and later, you need to open the file with a mystic incantation: csvfile = open(filehandle, newline='') > sample = csvfile.read( 1024 )# Grab a sample > csvfile.seek( 0 ) > dialect = csv.Sniffer().sniff(sample) # Check for file format with > sniffer. > csvfile = csv.reader( csvfile, dialect ) Use: csvfile = csv.reader(csvfile, dialect=dialect) dialect is a keyword argument. > if csv.Sniffer().has_header( sample ): #if there is a header > colnames = csvfile.next() # label columns from first line > datalist = list( csvfile ) # append data to a list Do you really need to use the Sniffer? You'll probably be better off -- Neil Cerutti *** You found a dead moose-rat. You sell the hide for $200. ***
From: Neil Cerutti on 3 Jun 2010 15:21 On 2010-06-03, Neil Cerutti <neilc(a)norwich.edu> wrote: > Do you really need to use the Sniffer? You'll probably be better > off... ....defining your own dialect based on what you know to be the file format. -- Neil Cerutti
From: Carlos Grohmann on 3 Jun 2010 15:26 Thanks for your prompt response, Neil. > That data doesn't appear to be csv worthy. Why not use str.split > or str.partition? Well, I should have said that this is one kind of data. The function is part of a larger app, and there is the possibility that someone uses headers in the data files, or some other field separator, so I tried to make it more universal. > In Python 2.6 and earlier, you need to open the file in binary > mode. I tried that, no changes > Use: > csvfile = csv.reader(csvfile, dialect=dialect) > dialect is a keyword argument. thanks for pointing that out.it stopped the errors when there s only one data line, but it still can't get the values for that line Carlos
From: Neil Cerutti on 3 Jun 2010 16:18
On 2010-06-03, Carlos Grohmann <carlos.grohmann(a)gmail.com> wrote: >> Use: >> ? ?csvfile = csv.reader(csvfile, dialect=dialect) >> dialect is a keyword argument. > > thanks for pointing that out.it stopped the errors when there s > only one data line, but it still can't get the values for that > line Is it possible your data is ill-formed in that case? If it's lacking a line-end, I don't know what should happen. -- Neil Cerutti |