Prev: reading a bunch of data files with non-consecutive file names
Next: Simscape>Mechanical Source
From: Anthony Hopf on 29 May 2010 12:30
Latest problem....

I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.

S = [1 2 3 0 0 0 0 0 0
4 5 6 0 0 0 0 0 0
7 8 9 10 11 12 13 14 0
15 16 17 18 19 20 21 22 23]

I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!

S_s = sparse(S);%<-- Much less memory (in the above case there isn't much savings)

Is there any way address S_s row wise to grab out the vectors I need without turning it back into a full matrix?

I am trying to figure out if I can use Bruno Luong's Sparse Sub Access, but I don't have su access to the server I am using to test it out...

Thanks in advance
From: Bruno Luong on 29 May 2010 12:43
"Anthony Hopf" <anthony.hopf(a)gmail.com> wrote in message <htrfev$fo7$1(a)fred.mathworks.com>...
> Latest problem....
>
> I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.
>
> S = [1 2 3 0 0 0 0 0 0
> 4 5 6 0 0 0 0 0 0
> 7 8 9 10 11 12 13 14 0
> 15 16 17 18 19 20 21 22 23]
>
> I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!
>
> S_s = sparse(S);%<-- Much less memory (in the above case there isn't much savings)
>
> Is there any way address S_s row wise to grab out the vectors I need without turning it back into a full matrix?

The following command will return the non-zeros elements of matrix:

nonzeros(S_s(row,:))

I strongly recommend to transpose your matrix so that to grab the index by column Due to the way sparse matrix organized in the computer memory, access by column is much faster.

AFAICS, you don not need my sparse access tool.

Bruno
From: Anthony Hopf on 29 May 2010 12:58
"Bruno Luong" <b.luong(a)fogale.findmycountry> wrote in message <htrg6p$2h3$1(a)fred.mathworks.com>...
> "Anthony Hopf" <anthony.hopf(a)gmail.com> wrote in message <htrfev$fo7$1(a)fred.mathworks.com>...
> > Latest problem....
> >
> > I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.
> >
> > S = [1 2 3 0 0 0 0 0 0
> > 4 5 6 0 0 0 0 0 0
> > 7 8 9 10 11 12 13 14 0
> > 15 16 17 18 19 20 21 22 23]
> >
> > I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!
> >
> > S_s = sparse(S);%<-- Much less memory (in the above case there isn't much savings)
> >
> > Is there any way address S_s row wise to grab out the vectors I need without turning it back into a full matrix?
>
> The following command will return the non-zeros elements of matrix:
>
> nonzeros(S_s(row,:))
>
> I strongly recommend to transpose your matrix so that to grab the index by column Due to the way sparse matrix organized in the computer memory, access by column is much faster.
>
> AFAICS, you don not need my sparse access tool.
>
> Bruno

Bruno,

That is a solid!! And you are right, transpose and it screams.

Thank you again!!

Anthony
From: Matt J on 29 May 2010 13:49
"Anthony Hopf" <anthony.hopf(a)gmail.com> wrote in message <htrfev$fo7$1(a)fred.mathworks.com>...
> Latest problem....
>
> I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.
>
> S = [1 2 3 0 0 0 0 0 0
> 4 5 6 0 0 0 0 0 0
> 7 8 9 10 11 12 13 14 0
> 15 16 17 18 19 20 21 22 23]
>
> I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!
==================

Why not construct S as sparse logical with each row of S (or column as Bruno recommended), consisting of logical indices instead of linear ones?

That way the sparse matrix would consume even less memory (because it's type logical instead of double). Plus, you wouldn't have to sift out the zeros when indexing, because sparse logical indexing already does that for you under the hood.
From: Anthony Hopf on 29 May 2010 14:10
"Matt J " <mattjacREMOVE(a)THISieee.spam> wrote in message <htrk32$2vm$1(a)fred.mathworks.com>...
> "Anthony Hopf" <anthony.hopf(a)gmail.com> wrote in message <htrfev$fo7$1(a)fred.mathworks.com>...
> > Latest problem....
> >
> > I have a matrix NxM that is N indexing vectors. Some of these vectors have very few index values, but they all have to be M long (the largest number of index values) so they are filled with zeros. Turns out I only utilize about 3% of the matrix to valid index values so I thought it would be a "good" idea to take advantage of sparse matrices to save a ton of memory, the MxN matrix can be over a gig.
> >
> > S = [1 2 3 0 0 0 0 0 0
> > 4 5 6 0 0 0 0 0 0
> > 7 8 9 10 11 12 13 14 0
> > 15 16 17 18 19 20 21 22 23]
> >
> > I would access each vector by S(1,:) and would have to cull out the 0's... but it would be great to just get the nonzeros!!
> ==================
>
> Why not construct S as sparse logical with each row of S (or column as Bruno recommended), consisting of logical indices instead of linear ones?
>
> That way the sparse matrix would consume even less memory (because it's type logical instead of double). Plus, you wouldn't have to sift out the zeros when indexing, because sparse logical indexing already does that for you under the hood.

Matt, that is a very good point. but I use the index values to pick out points in a 3d xyz data structure, add them and replace the value in the 3d xyz matrix, these index values, as noted above, are not uniform from spot to spot (I am simulating a beam sampling space so the number of valid points increase with range from the sensor).

I'll think about it, maybe I can apply this technique... Thanks again!

Anthony
 |  Next  |  Last
Pages: 1 2 3 4 5 6
Prev: reading a bunch of data files with non-consecutive file names
Next: Simscape>Mechanical Source