Dealing with large variables
in [Matlab]
|
From: Dan on 11 Aug 2010 08:44 I'm having some trouble getting MATLAB to handle some relatively simple processes with large variables. The simplified version of my code is as follows: ----------------------------------------------- %Pre-allocate Matricies (NumFiles could be as large as 500,000) x = zeros(NumFiles,351); R = zeros(NumFiles,100); c = zeros(351,100); %Populate x & R matricies ....STUFF %Perform Data Fitting [i,NumCoefs] = size(R); for i = 1:NumCoefs c(:,i) = x\R(:,i); end --------------------------------------------------- This code works OK if NumFiles < ~140K, but runs out of memory during preallocation of the 'x' variable if I try to use my full dataset. I've tried preallocating x,R & c as single precision (ie. x = zeros(NumFiles,351,'single'), which helps the memory issue, but then the solution to 'c' isn't even close to what I get using double precision. Any suggestions? Thanks, Dan
From: us on 11 Aug 2010 08:51 On Aug 11, 2:44 pm, "Dan " <dti...(a)stackpoleengineering.com> wrote: > I'm having some trouble getting MATLAB to handle some relatively simple processes with large variables. The simplified version of my code is as follows: > ----------------------------------------------- > %Pre-allocate Matricies (NumFiles could be as large as 500,000) > x = zeros(NumFiles,351); > R = zeros(NumFiles,100); > c = zeros(351,100); > > %Populate x & R matricies > ...STUFF > > %Perform Data Fitting > [i,NumCoefs] = size(R); > for i = 1:NumCoefs > c(:,i) = x\R(:,i); > end > --------------------------------------------------- > This code works OK if NumFiles < ~140K, but runs out of memory during preallocation of the 'x' variable if I try to use my full dataset. I've tried preallocating x,R & c as single precision (ie. x = zeros(NumFiles,351,'single'), which helps the memory issue, but then the solution to 'c' isn't even close to what I get using double precision. Any suggestions? > > Thanks, > Dan frankly: none but to reconsider your computational approach/engine... us
From: Dan on 11 Aug 2010 15:02 Fair enough... Any suggestions on how to do that? --Dan "Dan " <dtiley(a)stackpoleengineering.com> wrote in message <i3u5v9$qmc$1(a)fred.mathworks.com>... > I'm having some trouble getting MATLAB to handle some relatively simple processes with large variables. The simplified version of my code is as follows: > ----------------------------------------------- > %Pre-allocate Matricies (NumFiles could be as large as 500,000) > x = zeros(NumFiles,351); > R = zeros(NumFiles,100); > c = zeros(351,100); > > %Populate x & R matricies > ...STUFF > > %Perform Data Fitting > [i,NumCoefs] = size(R); > for i = 1:NumCoefs > c(:,i) = x\R(:,i); > end > --------------------------------------------------- > This code works OK if NumFiles < ~140K, but runs out of memory during preallocation of the 'x' variable if I try to use my full dataset. I've tried preallocating x,R & c as single precision (ie. x = zeros(NumFiles,351,'single'), which helps the memory issue, but then the solution to 'c' isn't even close to what I get using double precision. Any suggestions? > > Thanks, > Dan
From: Matt J on 11 Aug 2010 15:22 "Dan " <dtiley(a)stackpoleengineering.com> wrote in message <i3u5v9$qmc$1(a)fred.mathworks.com>... > --------------------------------------------------- > This code works OK if NumFiles < ~140K, but runs out of memory during preallocation of the 'x' variable if I try to use my full dataset. I've tried preallocating x,R & c as single precision (ie. x = zeros(NumFiles,351,'single'), which helps the memory issue, but then the solution to 'c' isn't even close to what I get using double precision. Any suggestions? =============== And you can't represent these in sparse type?
From: Walter Roberson on 11 Aug 2010 15:28 Dan wrote: > Fair enough... Any suggestions on how to do that? > > --Dan > > > "Dan " <dtiley(a)stackpoleengineering.com> wrote in message > <i3u5v9$qmc$1(a)fred.mathworks.com>... >> I'm having some trouble getting MATLAB to handle some relatively >> simple processes with large variables. The simplified version of my >> code is as follows: >> ----------------------------------------------- >> %Pre-allocate Matricies (NumFiles could be as large as 500,000) >> x = zeros(NumFiles,351); >> R = zeros(NumFiles,100); >> c = zeros(351,100); >> >> %Populate x & R matricies >> ...STUFF >> >> %Perform Data Fitting >> [i,NumCoefs] = size(R); >> for i = 1:NumCoefs >> c(:,i) = x\R(:,i); >> end >> --------------------------------------------------- >> This code works OK if NumFiles < ~140K, but runs out of memory during >> preallocation of the 'x' variable if I try to use my full dataset. >> I've tried preallocating x,R & c as single precision (ie. x = >> zeros(NumFiles,351,'single'), which helps the memory issue, but then >> the solution to 'c' isn't even close to what I get using double >> precision. Any suggestions? Just tossing out some ideas: - what if you try storing x and R as single precision but use double(x) \ double(R(:,i)) - what if you use cell arrays? As R is used column by column, it could be converted to a cell array of columns, which would reduce the need for contiguous memory - what is involved in building R ? Since you use only a column of it at a time, could you build R just a column at a time? - as the right-hand side of the \ operator is a single column vector at a time, is the implication that you are doing a least-squared fit? If so, then I wonder if there are algorithms for "chunking" the least-squared calculation, building up partial results over subsets of x ? That might require a pre-pass to subtract the mean, but calculation of the mean is something that you should be able to "chunk".
|
Next
|
Last
Pages: 1 2 Prev: GET PAID FOR WORKING ONLINE $100 ON YOUR FIRST DAY Next: fitting poins to a sing function |