Prev: how to index ans??
Next: Converting discrete (categorical) variable to Dichotomous variable?
From: us on 31 May 2010 08:37
"megha bhatt" <mubhatt19(a)yahoo.co.in> wrote in message <hu0a0m$dp0$1(a)fred.mathworks.com>...
> Hi.
> I have a huge data file with strings, numbers, and header. I want to read this file in MATLAB. If I try to read the file at once I get the memory full error message so I am trying to read the file in segments.
> My code to do so is :
> block_size = 1000;
> file_id = fopen('myfile.dat');
> format = '%d %d %s %f %f %f %f %f %f %f %f';
> while ~feof(file_id)
> test = textscan(file_id, format, block_size,'delimiter', ',','HeaderLines', 1);
> end
>
> The values I am getting at last are only from last segment of the reading, I am not getting how I can import the full file in MATLAB.
> Please help me to understand how I should read a huge file in MATLAB.
> Thanks,
> Megha

well... in your loop(?), you assign each new block to the same var TEST...
now: think...

us
From: dpb on 31 May 2010 08:42
megha bhatt wrote:
> Hi. I have a huge data file with strings, numbers, and header. I want to
> read this file in MATLAB. If I try to read the file at once I get the
> memory full error message so I am trying to read the file in segments.
> My code to do so is :

block_size = 1000;
> file_id = fopen('myfile.dat');
> format = '%d %d %s %f %f %f %f %f %f %f %f';
while ~feof(file_id)
> test = textscan(file_id, format, block_size, ...
> 'delimiter', ',', 'HeaderLines', 1);
> end
>
> The values I am getting at last are only from last segment of the
> reading,

Well, of course they are; you overwrite the array "test" on each pass
thru the loop this way.

I am not getting how I can import the full file in MATLAB.
> Please help me to understand how I should read a huge file in MATLAB.

Well, just how large is the file? If you ran out of memory reading it
in one swell foop, why would you expect it to take any less memory if
you tried to read the whole thing but in pieces?

The better route would be to try to do your processing in pieces inside
the above loop if possible. You could certainly hold more than 1000
entries at a time unless the strings were extremely long, but still
physical memory is physical memory and at some point you'll exhaust what
you have.

Alternatively, if you can do w/o all the variables at one time (fewer
columns) or decimate by row or some other reduction method those would
be possible alternatives. Or, find a machine w/ more capacity...

--
 | 
Pages: 1
Prev: how to index ans??
Next: Converting discrete (categorical) variable to Dichotomous variable?