opening large files with fopen/fread [Matlab]

Prev: Khoros Visualization image ( .xv format) not recognized in MATLAB!!
Next: Voronoi Polygon Areas

From: jimmy mac on 7 Apr 2010 22:38

Im trying to open some medium size files with fopen/fread.
From what i can see, people are claiming that even with 32 bit windows, you can open a 2 gigabyte file.

Im using two different computers, each with 8 gb of ram. One is windows 7, 64 bit, and vista 64 bit with matlab 2009 and 2008 respectively. The windows 7 machine is giving and out of memory error while the vista 64 machine is just locking up for 45 minutes until i kill it. The file I am trying to open is just 1gb is size with 16bit data. If it matters, the files are from daq device that samples at 100mhtz with 14 bit resolution and they get stored at 16bit. I would really like to open a 2-3 gigabyte file..

According to this older document (pre vista), i dont understand why im having these problems.
http://www.mathworks.com/support/tech-notes/1100/1106.html

I can break up the data if need be, but was under the impression this would work.
fid = fopen(filename,'r');
rf_d1 = double(fread(fid, 'int16')); % read at 14 bit as 16

From: Walter Roberson on 7 Apr 2010 22:55

jimmy mac wrote:
> Im trying to open some medium size files with fopen/fread.
> From what i can see, people are claiming that even with 32 bit windows,
> you can open a 2 gigabyte file.

> Im using two different computers, each with 8 gb of ram. One is windows
> 7, 64 bit, and vista 64 bit with matlab 2009 and 2008 respectively.

Please use cpu() and computer() to check to be sure you have a 64 bit version
of Matlab installed, and use memory() to see what the largest array size is
that you have available.

> fid = fopen(filename,'r');
> rf_d1 = double(fread(fid, 'int16')); % read
> at 14 bit as 16

If you happened to only have a 32 bit version of Matlab installed, the above
code is going to give you problems. You are attempting to read the entire 1 Gb
file (which is going to be roughly half a billion 2 byte samples) and you are
converting the data read into double precision... which is going to require
about half a billion 8 byte values, for a total of about 4 Gb. No 32 bit
Matlab version can handle 4 Gb arrays.

From: Rune Allnor on 8 Apr 2010 04:11

On 8 apr, 04:38, "jimmy mac" <jimmyi...(a)gmail.com> wrote:
> Im trying to open some medium size files with fopen/fread.
> From what i can see, people are claiming that even with 32 bit windows, you can open a 2 gigabyte file.

One can *open* it, but not necessarily *load* all of it.

> Im using two different computers, each with 8 gb of ram. One is windows 7, 64 bit, and vista 64 bit with matlab 2009 and 2008 respectively. The windows 7 machine is giving and out of memory error while the vista 64 machine is just locking up for 45 minutes until i kill it. The file I am trying to open is just 1gb is size with 16bit data. If it matters, the files are from daq device that samples at 100mhtz with 14 bit resolution and they get stored at 16bit. I would really like to open a 2-3 gigabyte file...

Open the file, load only part of it.

> I can break up the data if need be, but was under the impression this would work.
> fid = fopen(filename,'r');
> rf_d1 = double(fread(fid, 'int16'));

This works *provided* there is enough memory available to
contain the data. If there isn't, you will get an out-of-memory
error. "Memory" here means what is available to matlab. If you
have a multicore CPU, each CPU will have a fraction of the
total memory. The core will divide its available RAM between the
processes that run on the core. The individual program will
be limited by the maximum continuous segment of memory available
to it. Programs need some 2-5 times the space occupied by variables
for internal temporary variables.

And so on and so forth.

All in all, the maximum available memory segment might be
considerably smaller than the total amount of memory.

Again, the standard way to approach this is is to open the file,
and only load as much as you can conveniently handle at any one
time.

Rune

From: us on 8 Apr 2010 04:20

"jimmy mac" <jimmyinct(a)gmail.com> wrote in message <hpjfic$rgq$1(a)fred.mathworks.com>...
> Im trying to open some medium size files with fopen/fread.
> From what i can see, people are claiming that even with 32 bit windows, you can open a 2 gigabyte file.
>
> Im using two different computers, each with 8 gb of ram. One is windows 7, 64 bit, and vista 64 bit with matlab 2009 and 2008 respectively. The windows 7 machine is giving and out of memory error while the vista 64 machine is just locking up for 45 minutes until i kill it. The file I am trying to open is just 1gb is size with 16bit data. If it matters, the files are from daq device that samples at 100mhtz with 14 bit resolution and they get stored at 16bit. I would really like to open a 2-3 gigabyte file..
>
> According to this older document (pre vista), i dont understand why im having these problems.
> http://www.mathworks.com/support/tech-notes/1100/1106.html
>
> I can break up the data if need be, but was under the impression this would work.
> fid = fopen(filename,'r');
> rf_d1 = double(fread(fid, 'int16')); % read at 14 bit as 16

in addition to what was already said, you might consider

help memmapfile;

us

From: jim on 8 Apr 2010 10:54

| Next | Last
Pages: 1 2
Prev: Khoros Visualization image ( .xv format) not recognized in MATLAB!!
Next: Voronoi Polygon Areas