parsing a large blob of ascii data
in [Matlab]
|
Prev: PI
Next: Carriage return in mex files
From: Tracy on 9 Jun 2010 15:55 Walter Roberson <roberson(a)hushmail.com> wrote in message <huom2d$5gb$1(a)canopus.cc.umanitoba.ca>... > Tracy wrote: > > "Tracy " <tracy(a)stratalight.com> wrote in message > > <huoj4q$dv5$1(a)fred.mathworks.com>... > >> sure - and thanks for the quick response > >> > >> The data looks like this - i want all of the columns to be strings. > >> this is just 1 row. there can be up to 64K rows. > >> > >> DataPat CRIT 01/01/70:00:00:28 > >> thunder.cpp 0516 Thunder 0 Alignment failed 0x0 > > > > I should have mentioned that this is 6 columns which are fixed length - > > 8 chars > > 8 chars > > 32 chars > > 32 chars > > 6 chars > > 256 chars > > I tried that and i get empty strings. :-( K>> xx=textscan(data, '%8c\\0%8c\\0%32c\\03%2c\\%06c\\0%256c', 'Whitespace', '', 'Delimiter', ''); K>> xx xx = [0x8 char] [0x8 char] [0x32 char] [0x2 char] [0x6 char] [0x256 char] > > it's not apparent from the display that this is the case - they are null > > terminated. > > Null termination is usually a signal to end the string; I'm not surprised only > a single field was returned. > > Try something like this, > > textscan('%8c\0%8c\0%32c\0%32c\0%6c\0%256c', ... > 'Whitespace', '', 'Delimeter', '');
From: Walter Roberson on 9 Jun 2010 16:19 Tracy top-posted, making a mess of the flow of discussion: > Walter Roberson <roberson(a)hushmail.com> wrote in message >> > > I should have mentioned that this is 6 columns which are fixed >> length - >> > 8 chars >> > 8 chars >> > 32 chars >> > 32 chars >> > 6 chars >> > 256 chars >> > it's not apparent from the display that this is the case - they are >> null > terminated. >> Try something like this, >> >> textscan('%8c\0%8c\0%32c\0%32c\0%6c\0%256c', ... >> 'Whitespace', '', 'Delimeter', ''); > I tried that and i get empty strings. :-( > > > K>> xx=textscan(data, '%8c\\0%8c\\0%32c\\03%2c\\%06c\\0%256c', > 'Whitespace', '', 'Delimiter', ''); Ah, fooey, textscan() doesn't recognize \ constants like sscanf does. Okay, in that case, textscan('%8c%*c%8c%*c%32c%*c%32c%*c%6c%*c%256c'); The Whitespace and Delimiter do not need to be specified for this format.
From: Tracy on 9 Jun 2010 16:33 Walter Roberson <roberson(a)hushmail.com> wrote in message <huot4m$fs6$1(a)canopus.cc.umanitoba.ca>... > Tracy top-posted, making a mess of the flow of discussion: > > > Walter Roberson <roberson(a)hushmail.com> wrote in message > > >> > > I should have mentioned that this is 6 columns which are fixed > >> length - > >> > 8 chars > >> > 8 chars > >> > 32 chars > >> > 32 chars > >> > 6 chars > >> > 256 chars > > >> > it's not apparent from the display that this is the case - they are > >> null > terminated. > > >> Try something like this, > >> > >> textscan('%8c\0%8c\0%32c\0%32c\0%6c\0%256c', ... > >> 'Whitespace', '', 'Delimeter', ''); > > > I tried that and i get empty strings. :-( > > > > > > K>> xx=textscan(data, '%8c\\0%8c\\0%32c\\03%2c\\%06c\\0%256c', > > 'Whitespace', '', 'Delimiter', ''); > > Ah, fooey, textscan() doesn't recognize \ constants like sscanf does. Okay, in > that case, > > textscan('%8c%*c%8c%*c%32c%*c%32c%*c%6c%*c%256c'); > > The Whitespace and Delimiter do not need to be specified for this format. still empty cells :-( K>> xx=textscan(data, '%8c%*c%8c%*c%32c%*c%32c%*c%6c%*c%256c'); K>> xx xx = [0x8 char] [0x8 char] [0x32 char] [0x32 char] [0x6 char] [0x256 char] I have control over both sides of this interface. Would it fix the issue if i space padded the fields instead of null padded? Note that the last column is whatever is left on the row (so it will have spaces between words). I thought using fixed length fields would be super simple :-(
From: Walter Roberson on 9 Jun 2010 16:52 Tracy wrote: > still empty cells :-( > > K>> xx=textscan(data, '%8c%*c%8c%*c%32c%*c%32c%*c%6c%*c%256c'); > K>> xx > > xx = > [0x8 char] [0x8 char] [0x32 char] [0x32 char] [0x6 > char] [0x256 char] Very odd unless there are fewer than 8 characters available. Is data a string or a fid (file ID) returned from fopen()? textscan permits either in that position. Just before the call, double check class(data) size(data) even if you _think_ that it is an fid. If it is an fid, did you open for binary input ('r') or for text ('rt') ? If you opened for binary then you could fseek(data, 0, 'eof') and ftell(data) to find out how big the file is; this will not work for text files on Windows. > I have control over both sides of this interface. Would it fix the > issue if i space padded the fields instead of null padded? Maybe. Or you could use | or some other character you know does not appear, and then you could do the reading by textscan(data, '%s%s%s%s%s%s', 'Delimiter', '|'); > Note that > the last column is whatever is left on the row (so it will have spaces > between words). Possibly that is the problem, maybe. You could try using %[^\n] as the last format instead of %256c, provided that there is an end of line between the "lines".
From: Tracy on 9 Jun 2010 17:12
Walter Roberson <roberson(a)hushmail.com> wrote in message <huov1i$ijj$1(a)canopus.cc.umanitoba.ca>... > Tracy wrote: > > > still empty cells :-( > > > > K>> xx=textscan(data, '%8c%*c%8c%*c%32c%*c%32c%*c%6c%*c%256c'); > > K>> xx > > > > xx = > > [0x8 char] [0x8 char] [0x32 char] [0x32 char] [0x6 > > char] [0x256 char] > > Very odd unless there are fewer than 8 characters available. > > Is data a string or a fid (file ID) returned from fopen()? textscan permits > either in that position. Just before the call, double check > > class(data) > size(data) > > even if you _think_ that it is an fid. > > If it is an fid, did you open for binary input ('r') or for text ('rt') ? If > you opened for binary then you could fseek(data, 0, 'eof') and ftell(data) to > find out how big the file is; this will not work for text files on Windows. > > > I have control over both sides of this interface. Would it fix the > > issue if i space padded the fields instead of null padded? > > Maybe. Or you could use | or some other character you know does not appear, > and then you could do the reading by > > textscan(data, '%s%s%s%s%s%s', 'Delimiter', '|'); > > > Note that > > the last column is whatever is left on the row (so it will have spaces > > between words). > > Possibly that is the problem, maybe. You could try using %[^\n] as the last > format instead of %256c, provided that there is an end of line between the > "lines". actually "data" is a 1xM array of char that i have read from a tcpip connection. but i am getting closer. I added this stripped_data=strrep(data, char(0), ' '); celldata = textscan(stripped_data, '%8c%8c%32c%32c%6c%256c'); and then celldata (is a 1x6 cell) looks like this celldata = [56x8 char] [56x8 char] [56x32 char] [56x32 char] [56x6 char] [56x256 char] which is very close! But what i need is a matrix of cells (so i can give it to my table object using set(handles.log_table, 'Data', celldata); in this case i would want celldata to be a 6 column, 56 row matrix. Can you help me with this conversion? |