Prev: GET PAID FOR WORKING ONLINE $100 ON YOUR FIRST DAY
Next: fitting poins to a sing function
From: Dan on 12 Aug 2010 13:19
>
> And you can't represent these in sparse type?


I tried preallocating x, R & c as "sparse". It brought the memory usage down considerably, but a test run with a small dataset yielded significantly different results to my 'c' matrix. Here's what I did:
%Preallocate matrix sizes
x = sparse(NumFiles,351);
R = sparse(NumFiles,100);
c = sparse(351,100);

instead of:
%Preallocate matrix sizes
x = zeros(NumFiles,351);
R = zeros(NumFiles,100);
c = zeros(351,100);

Perhaps I have the wrong syntax?

--Dan
From: Dan on 12 Aug 2010 13:26
Walter Roberson <roberson(a)hushmail.com> wrote in message <i3utm1
> Just tossing out some ideas:
>
> - what if you try storing x and R as single precision but use
> double(x) \ double(R(:,i))
>
> - what if you use cell arrays? As R is used column by column, it could be
> converted to a cell array of columns, which would reduce the need for
> contiguous memory
>
> - what is involved in building R ? Since you use only a column of it at a
> time, could you build R just a column at a time?
>
> - as the right-hand side of the \ operator is a single column vector at a
> time, is the implication that you are doing a least-squared fit? If so, then I
> wonder if there are algorithms for "chunking" the least-squared calculation,
> building up partial results over subsets of x ? That might require a pre-pass
> to subtract the mean, but calculation of the mean is something that you should
> be able to "chunk".

Walter,
Your first suggestion seems to be working pretty well so far, using a smaller test dataset. The 'c' matrix solutions are very close (typically less than 1% difference) to that of having initiated the 'x' & 'R' matricies as double precision.

I like your idea about "chunking" the least-squared calculation (yes, you were right about this program preforming a least-squared regression). I'm not sure of a valid way to do that, however... not from a coding standpoint, but more from a standpoint of being mathematically equivalent. I'll try some things to see if I can find a way to do this.

Thanks for the help!
--Dan
From: Matt J on 12 Aug 2010 14:30
"Dan " <dtiley(a)stackpoleengineering.com> wrote in message <i41aec$ici$1(a)fred.mathworks.com>...
> >
> > And you can't represent these in sparse type?
>
>
> I tried preallocating x, R & c as "sparse". It brought the memory usage down considerably, but a test run with a small dataset yielded significantly different results to my 'c' matrix. Here's what I did:
> %Preallocate matrix sizes
> x = sparse(NumFiles,351);
> R = sparse(NumFiles,100);
> c = sparse(351,100);
==================

That doesn't have a meaningful pre-allocation effect, because sparse(M,N) doesn't tell MATLAB how many non-zero elements the matrix will hold. You really should be using the form sparse(i,j,s) to set up these matrices.

Also, assuming your data is sparse enough, there's no reason to be using a for-loop anymore, and hence no need to preallcoate c. You could just be doing c=x\R;

As for why you're getting different results for c, the pre-allocation method won't tell us anything about that. MATLAB's sparse solvers use different algorithms than the full solvers, so perhaps your problem is numerically sensitive.

Also, you should use isequal() to make sure the sparse and full versions of the matrices are indeed being populated equivalently.
First  |  Prev  | 
Pages: 1 2
Prev: GET PAID FOR WORKING ONLINE $100 ON YOUR FIRST DAY
Next: fitting poins to a sing function