Prev: Optimisation toolbox
Next: small for loop creating a multidimensional array
From: Adam Thibideau on 12 Jul 2010 18:41
I am using textscan() to import data from a large CSV file (about 12 Million rows x 4 cols). My problem is that I am running out of contiguous memory to store the resulting struct that I am creating.

Is there any way to increase the amount of contiguous memory available? I am launching matlab using the command prompt with "matlab -shield maximum" and when i use the "feature memstats" command, the largest contiguous block I am getting is 1500MB and the second largest is 900MB.

I am on a 32bit system with 3GB of physical RAM .

If there is nothing I can do to increase the contiguous memory, is there a work around for this??

Thanks!
From: us on 12 Jul 2010 18:58
"Adam Thibideau" <adam.thibideau(a)gmail.com> wrote in message <i1g5m0$di9$1(a)fred.mathworks.com>...
> I am using textscan() to import data from a large CSV file (about 12 Million rows x 4 cols). My problem is that I am running out of contiguous memory to store the resulting struct that I am creating.
>
> Is there any way to increase the amount of contiguous memory available? I am launching matlab using the command prompt with "matlab -shield maximum" and when i use the "feature memstats" command, the largest contiguous block I am getting is 1500MB and the second largest is 900MB.
>
> I am on a 32bit system with 3GB of physical RAM .
>
> If there is nothing I can do to increase the contiguous memory, is there a work around for this??
>
> Thanks!

what's the content of your data(?)...
- doubles
- char strings
- mixed stuff
how do you read your file(?)...

us
From: kk KKsingh on 12 Jul 2010 19:00
"Adam Thibideau" <adam.thibideau(a)gmail.com> wrote in message <i1g5m0$di9$1(a)fred.mathworks.com>...
> I am using textscan() to import data from a large CSV file (about 12 Million rows x 4 cols). My problem is that I am running out of contiguous memory to store the resulting struct that I am creating.
>
> Is there any way to increase the amount of contiguous memory available? I am launching matlab using the command prompt with "matlab -shield maximum" and when i use the "feature memstats" command, the largest contiguous block I am getting is 1500MB and the second largest is 900MB.
>
> I am on a 32bit system with 3GB of physical RAM .
>
> If there is nothing I can do to increase the contiguous memory, is there a work around for this??
>
> Thanks!

Set the swap space on your system to the maximum that can be utilized by
your operating system

On Linux, swap space can be changed by using the mkswap and swapon
commands. For more information on the above commands, type man
command_name at the Linux prompt.


Also try running it it matlab -nojvm mode, it saves some memory !

Hope that will help

Kumar
From: Adam Thibideau on 12 Jul 2010 19:23
"us " <us(a)neurol.unizh.ch> wrote in message <i1g6ls$efi$1(a)fred.mathworks.com>...
> "Adam Thibideau" <adam.thibideau(a)gmail.com> wrote in message <i1g5m0$di9$1(a)fred.mathworks.com>...
> > I am using textscan() to import data from a large CSV file (about 12 Million rows x 4 cols). My problem is that I am running out of contiguous memory to store the resulting struct that I am creating.
> >
> > Is there any way to increase the amount of contiguous memory available? I am launching matlab using the command prompt with "matlab -shield maximum" and when i use the "feature memstats" command, the largest contiguous block I am getting is 1500MB and the second largest is 900MB.
> >
> > I am on a 32bit system with 3GB of physical RAM .
> >
> > If there is nothing I can do to increase the contiguous memory, is there a work around for this??
> >
> > Thanks!
>
> what's the content of your data(?)...
> - doubles
> - char strings
> - mixed stuff
> how do you read your file(?)...
>
> us


It is four columns: string, double, string, float scanned in like this:

textmatrix = textscan(fid,'%s%d%s%f32',nrows,'Delimiter',',','HeaderLines','1');

where nrows is how many rows i take in at a time. usually i try to put 1million here.
From: Kimberly on 13 Jul 2010 10:32
"Adam Thibideau" <adam.thibideau(a)gmail.com> wrote in message <i1g84p$db6$1(a)fred.mathworks.com>...

> > >
> > > I am on a 32bit system with 3GB of physical RAM .
> > >
> > > If there is nothing I can do to increase the contiguous memory, is there a work around for this??
> > >

You perhaps could try memmapfile. It opens the file and, instead of reading the data into memory, it returns a buffer which looks AS IF the data is loaded into memory. Analogous to java's Java.nio.MappedByteBuffer, and a vast improvement fiddling around with a random access file.

Cheers,
Kimberly
 |  Next  |  Last
Pages: 1 2
Prev: Optimisation toolbox
Next: small for loop creating a multidimensional array