Prev: help me with frame2im
Next: indexing and reordering issue
From: Gerda Edelman on 20 Apr 2010 07:17
I have a problem concerning a large ascii file (1.2 GB), which looks like this:
29 lines of text data
512x640 pixelvalues of a frame
5 lines of text data
512x640 pixelvalues of a frame
5 lines of text data
512x640 pixelvalues of a frame
5 lines of text data
512x640 pixelvalues of a frame
etc.

If I use textscan to read this file my laptop runs out of memory. Using dlmread it goes fine for the first frames, but every next frame it takes longer to be read. I tried some other things, but mostly this resulted in memory problems. Does anybody know a resolution for this, which uses less memory?

Regards, Gerda
From: us on 20 Apr 2010 07:42
"Gerda Edelman" <gerdaedelman(a)yahoo.de> wrote in message <hqk2fg$bht$1(a)fred.mathworks.com>...
> I have a problem concerning a large ascii file (1.2 GB), which looks like this:
> 29 lines of text data
> 512x640 pixelvalues of a frame
> 5 lines of text data
> 512x640 pixelvalues of a frame
> 5 lines of text data
> 512x640 pixelvalues of a frame
> 5 lines of text data
> 512x640 pixelvalues of a frame
> etc.
>
> If I use textscan to read this file my laptop runs out of memory. Using dlmread it goes fine for the first frames, but every next frame it takes longer to be read. I tried some other things, but mostly this resulted in memory problems. Does anybody know a resolution for this, which uses less memory?
>
> Regards, Gerda

a hint:
- low level io functions...
- memory mapping the file...

help fopen;
help fseek;
help fread;
help fclose;
help memmapfile;

us
From: Rune Allnor on 20 Apr 2010 08:32
On 20 apr, 13:17, "Gerda Edelman" <gerdaedel...(a)yahoo.de> wrote:
> I have a problem concerning a large ascii file (1.2 GB), which looks like this:
> 29 lines of text data
> 512x640 pixelvalues of a frame
> 5 lines of text data
> 512x640 pixelvalues of a frame
> 5 lines of text data
> 512x640 pixelvalues of a frame
> 5 lines of text data
> 512x640 pixelvalues of a frame
> etc.
>
> If I use textscan to read this file my laptop runs out of memory. Using dlmread it goes fine for the first frames, but every next frame it takes longer to be read. I tried some other things, but mostly this resulted in memory problems. Does anybody know a resolution for this, which uses less memory?

Read only as much of the data that you need. This might mean
that you have to write your own parser, but that's not too hard
with the indicated file format.

Rune
 | 
Pages: 1
Prev: help me with frame2im
Next: indexing and reordering issue