Prev: Command to display what Matlab is doing?
Next: Muscle Wrapping over an cylinder - Shortest path
From: Thibault Daoulas on 4 Jun 2010 06:29
"Thibault Daoulas" <thibault.daoulas(a)gmail.com> wrote in message <huainb$cfp$1(a)fred.mathworks.com>...
> "Bruno Luong" <b.luong(a)fogale.findmycountry> wrote in message <huadq5$lbl$1(a)fred.mathworks.com>...
> > "Steve Amphlett" <Firstname.Lastname(a)Where-I-Work.com> wrote in message <hu8qjp$1hu$1(a)fred.mathworks.com>...
> >
> > >
> > > [m,n]=size(x);
> > > x(m*2,1)=0;
> >
> > Or with a single-line command (should not start from empty though)
> >
> > x(2*end,end)=0;
> >
> > Bruno

Hi again, sorry for the repetitive posts, but yes, finally the actual problem are the operations on the matrix, which take most of the time, and get more costly as the matrix get larger. I just did not expect that as big.
Up to the profiler then, what cost a lot is operations such as :
- [val index] = min(abs(buffer(:, 1)-x));
which looks for the closest value to x in buffer(:, 1)
- buffer(1:stepBuffer-1, :) = buffer(2:stepBuffer, :);
which updates the buffer (head taken out, new value tailed, indexes updated)
- if(stepBuffer>(5/6)*length(buffer(:, 1)))
buffer=cat(1, buffer, zeros(length(buffer(:, 1)), 3));
end
which extends the buffer size when needed

And the 3 of them consume 60%, while built-ins, overhead, etc. consume close to the 40% remaining.

I do not think there's a way to work around that...
From: dpb on 4 Jun 2010 09:41
Thibault Daoulas wrote:
....

> process, hence declared as persistent, in Matlab terms. I suspected that
> to be the cause of such slow handling, compared to for instance dealing
> with it as a global variable, though I'm not really aware of what this
> difference implies in the memory.

> As for the code, it's that kind of things :
>
> main.m :
> for ii=length(X)
> %do a bunch of things
> my_slow_function(X, a(ii), b(ii), c(ii));
> end
>
> my_slow_funcion.m :
> function result = my_slow_funcion(A, B, C)
> persistent my_matrix;
> %fill my_matrix with A, B, C
> my_matrix = cat(1, my_matrix, [A B C]);
> %compute stuff from the whole data contained in my_matrix
> getOldData = min(abs(my_matrix(:, 1)-A/1000))
> getOldMean = mean(my_matrix(A:B));
> result = [getOldData getOldMean];
>
>
> So yeah, that's the idea, and I'm actually assigning data linearly, as
> only 3 numbers are appened at each execution of my_slow_function. What
> takes more time obviously with the size is that :
> - min(abs(my_matrix(:, 1)-A/1000))
> But I don't think this is the cause of the problem. Apparently it's
> keeping my_matrix up to date, but again, I'm doubtful...

PERSISTENT vis a vis GLOBAL would make no difference in speed imo altho
it would be quite easy in this code to test that assumption empirically
it would appear.

I haven't had my morning coffee yet so I'm not thinking straight but
something is nagging me on the code but I can't put finger on it.
Somehow it looks to me this out to be simplifiable grossly to avoid the
concatenation but not clear enough either owing to time 'o day here or
the trimming to a small sample or some of both... :)

As for the doubt as to what's the bottleneck, use the profiler and
determine the splits on timing for sure and post the results.
Meanwhile, the coffee _is_ brewing...maybe there's hope. :)

--
From: Steven Lord on 4 Jun 2010 09:55

"Thibault Daoulas" <thibault.daoulas(a)gmail.com> wrote in message
news:huainb$cfp$1(a)fred.mathworks.com...
> "Bruno Luong" <b.luong(a)fogale.findmycountry> wrote in message
> <huadq5$lbl$1(a)fred.mathworks.com>...
>> "Steve Amphlett" <Firstname.Lastname(a)Where-I-Work.com> wrote in message
>> <hu8qjp$1hu$1(a)fred.mathworks.com>...
>>
>> >
>> > [m,n]=size(x);
>> > x(m*2,1)=0;
>>
>> Or with a single-line command (should not start from empty though)
>>
>> x(2*end,end)=0;
>>
>> Bruno
>
> Yep, that makes sense. The persistent thing is not plumbing the whole
> thing indeed, but rather the operations on the matrix. For instance among
> others I calculate a mean each time, and writing a small example, I found
> out this was at least one of the bad stuff that make the execution time
> sink..
> I shoudn't be suprised about that, but that is quite annoying, as I need
> to keep this history of means... Any suggestion on that big boys ? :)
>
> %----------------------------------------
> tic
> buffSizeM=10000;
> bufM=zeros(buffSizeM, 3);
>
> for ii=1:buffSizeM
> bufM(ii, :)=[ii ii ii/3];
> mean(bufM(1:ii, 2));
> end
> toc

Two things:

1) If that's the form of your bufM, you can create it without FOR loops.

z = (1:buffSizeM).';
bufM = [z, z, z/3];

2) Why are you computing the MEAN of a portion of the second column of bufM
and then throwing that result away? If you really want the MEAN of that
second column, there's no need to create bufM (unless you also need that:

% Remember, mean(x) is sum(x)./numel(x)
% so sum(x(1:ii)) in this case is the (ii)th element of
% cumsum(x), and numel(x(1:ii)) is ii which is
% the (ii)th element of z due to how I created it
thesum = cumsum(z);
themean = thesum./z;

When I got rid of the FOR loop and the buffSizeM calls to MEAN, this
executed VERY quickly.

function timingtest72


for buffSizeM = [10000 20000 50000 100000]

[b1, m1] = buffSize1(buffSizeM);

[b2, m2] = buffSize2(buffSizeM);

[b3, m3] = buffSize1(buffSizeM);

fprintf(['Difference in bufM between 1 and 2 was %g.\n', ...

'Difference in themean between 1 and 2 was %g.\n'], ...

norm(b1-b2), norm(m1-m2));

fprintf(['Difference in bufM between 2 and 3 was %g.\n', ...

'Difference in themean between 2 and 3 was %g.\n'], ...

norm(b2-b3), norm(m2-m3));

fprintf(['Difference in bufM between 3 and 1 was %g.\n', ...

'Difference in themean between 3 and 1 was %g.\n'], ...

norm(b3-b1), norm(m3-m1));

end




function [bufM, themean] = buffSize1(buffSizeM)

tic

bufM=zeros(buffSizeM, 3);

themean = zeros(buffSizeM, 1);

for ii=1:buffSizeM

bufM(ii, :)=[ii ii ii/3];

themean(ii) = mean(bufM(1:ii, 2));

end

t = toc;

fprintf('Buffer size %d, buffSize1, elapsed time is %g.\n', buffSizeM, t);





function [bufM, themean] = buffSize2(buffSizeM)

tic

z = (1:buffSizeM).';

bufM = [z, z, z/3];


thesum = cumsum(z);

themean = thesum./z;

t = toc;

fprintf('Buffer size %d, buffSize2, elapsed time is %g.\n', buffSizeM, t);


--
Steve Lord
slord(a)mathworks.com
comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
To contact Technical Support use the Contact Us link on
http://www.mathworks.com


From: Thibault Daoulas on 4 Jun 2010 10:22
dpb <none(a)non.net> wrote in message <huavvd$n1s$1(a)news.eternal-september.org>...
> Thibault Daoulas wrote:
> ...
>
> > process, hence declared as persistent, in Matlab terms. I suspected that
> > to be the cause of such slow handling, compared to for instance dealing
> > with it as a global variable, though I'm not really aware of what this
> > difference implies in the memory.
>
> > As for the code, it's that kind of things :
> >
> > main.m :
> > for ii=length(X)
> > %do a bunch of things
> > my_slow_function(X, a(ii), b(ii), c(ii));
> > end
> >
> > my_slow_funcion.m :
> > function result = my_slow_funcion(A, B, C)
> > persistent my_matrix;
> > %fill my_matrix with A, B, C
> > my_matrix = cat(1, my_matrix, [A B C]);
> > %compute stuff from the whole data contained in my_matrix
> > getOldData = min(abs(my_matrix(:, 1)-A/1000))
> > getOldMean = mean(my_matrix(A:B));
> > result = [getOldData getOldMean];
> >
> >
> > So yeah, that's the idea, and I'm actually assigning data linearly, as
> > only 3 numbers are appened at each execution of my_slow_function. What
> > takes more time obviously with the size is that :
> > - min(abs(my_matrix(:, 1)-A/1000))
> > But I don't think this is the cause of the problem. Apparently it's
> > keeping my_matrix up to date, but again, I'm doubtful...
>
> PERSISTENT vis a vis GLOBAL would make no difference in speed imo altho
> it would be quite easy in this code to test that assumption empirically
> it would appear.
>
> I haven't had my morning coffee yet so I'm not thinking straight but
> something is nagging me on the code but I can't put finger on it.
> Somehow it looks to me this out to be simplifiable grossly to avoid the
> concatenation but not clear enough either owing to time 'o day here or
> the trimming to a small sample or some of both... :)
>
> As for the doubt as to what's the bottleneck, use the profiler and
> determine the splits on timing for sure and post the results.
> Meanwhile, the coffee _is_ brewing...maybe there's hope. :)
>
> --

Hehe, well apparently your coffee had its effects on me.
I really had to run it on various data sizes, to actually find out what was good/bad, up to this. I was maybe gaining 1 minute on small samples with one implementation, but wasting 15 minutes on larger sizes and so on...
Indeed, you were right too, persistent vs global, tight game, same order of time consumption.
So what was wrong ? The time wasted by the actual operations on the elements of the matrix, although they were not philosophical operations :) ... So yep, the operations on that size was the problem.
No way to change the operations, so I tried to reduce the size of the matrix, by getting rid of the superfulous data, and staying with the smallest matrix as possible.
WRONG : the operation itself of updating the values in the matrix, and keeping it "small" took most of the time.

So I simply combined both good sides of these solutions : preallocation of the biggest matrix as possible, not not touch anything about its size etc., and restrict the domain of the operations on that matrix.

Preallocation and profiler are my best friends now.

Moral of the story : take as much space as you can, but use only what you need !

Thank you dpb, and all you guys !
From: Thibault Daoulas on 4 Jun 2010 10:28
"Steven Lord" <slord(a)mathworks.com> wrote in message <hub0k6$140$1(a)fred.mathworks.com>...
>
> "Thibault Daoulas" <thibault.daoulas(a)gmail.com> wrote in message
> news:huainb$cfp$1(a)fred.mathworks.com...
> > "Bruno Luong" <b.luong(a)fogale.findmycountry> wrote in message
> > <huadq5$lbl$1(a)fred.mathworks.com>...
> >> "Steve Amphlett" <Firstname.Lastname(a)Where-I-Work.com> wrote in message
> >> <hu8qjp$1hu$1(a)fred.mathworks.com>...
> >>
> >> >
> >> > [m,n]=size(x);
> >> > x(m*2,1)=0;
> >>
> >> Or with a single-line command (should not start from empty though)
> >>
> >> x(2*end,end)=0;
> >>
> >> Bruno
> >
> > Yep, that makes sense. The persistent thing is not plumbing the whole
> > thing indeed, but rather the operations on the matrix. For instance among
> > others I calculate a mean each time, and writing a small example, I found
> > out this was at least one of the bad stuff that make the execution time
> > sink..
> > I shoudn't be suprised about that, but that is quite annoying, as I need
> > to keep this history of means... Any suggestion on that big boys ? :)
> >
> > %----------------------------------------
> > tic
> > buffSizeM=10000;
> > bufM=zeros(buffSizeM, 3);
> >
> > for ii=1:buffSizeM
> > bufM(ii, :)=[ii ii ii/3];
> > mean(bufM(1:ii, 2));
> > end
> > toc
>
> Two things:
>
> 1) If that's the form of your bufM, you can create it without FOR loops.
>
> z = (1:buffSizeM).';
> bufM = [z, z, z/3];
>
> 2) Why are you computing the MEAN of a portion of the second column of bufM
> and then throwing that result away? If you really want the MEAN of that
> second column, there's no need to create bufM (unless you also need that:
>
> % Remember, mean(x) is sum(x)./numel(x)
> % so sum(x(1:ii)) in this case is the (ii)th element of
> % cumsum(x), and numel(x(1:ii)) is ii which is
> % the (ii)th element of z due to how I created it
> thesum = cumsum(z);
> themean = thesum./z;
>
> When I got rid of the FOR loop and the buffSizeM calls to MEAN, this
> executed VERY quickly.
>
> function timingtest72
>
>
> for buffSizeM = [10000 20000 50000 100000]
>
> [b1, m1] = buffSize1(buffSizeM);
>
> [b2, m2] = buffSize2(buffSizeM);
>
> [b3, m3] = buffSize1(buffSizeM);
>
> fprintf(['Difference in bufM between 1 and 2 was %g.\n', ...
>
> 'Difference in themean between 1 and 2 was %g.\n'], ...
>
> norm(b1-b2), norm(m1-m2));
>
> fprintf(['Difference in bufM between 2 and 3 was %g.\n', ...
>
> 'Difference in themean between 2 and 3 was %g.\n'], ...
>
> norm(b2-b3), norm(m2-m3));
>
> fprintf(['Difference in bufM between 3 and 1 was %g.\n', ...
>
> 'Difference in themean between 3 and 1 was %g.\n'], ...
>
> norm(b3-b1), norm(m3-m1));
>
> end
>
>
>
>
> function [bufM, themean] = buffSize1(buffSizeM)
>
> tic
>
> bufM=zeros(buffSizeM, 3);
>
> themean = zeros(buffSizeM, 1);
>
> for ii=1:buffSizeM
>
> bufM(ii, :)=[ii ii ii/3];
>
> themean(ii) = mean(bufM(1:ii, 2));
>
> end
>
> t = toc;
>
> fprintf('Buffer size %d, buffSize1, elapsed time is %g.\n', buffSizeM, t);
>
>
>
>
>
> function [bufM, themean] = buffSize2(buffSizeM)
>
> tic
>
> z = (1:buffSizeM).';
>
> bufM = [z, z, z/3];
>
>
> thesum = cumsum(z);
>
> themean = thesum./z;
>
> t = toc;
>
> fprintf('Buffer size %d, buffSize2, elapsed time is %g.\n', buffSizeM, t);
>
>
> --
> Steve Lord
> slord(a)mathworks.com
> comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
> To contact Technical Support use the Contact Us link on
> http://www.mathworks.com
>
Hi Steven,

This was just an example, for which i did not care about the result, just to illustrate the kind of operations within the loop, so concretely I'm not computing that mean, it's a little bit trickier, and the whole thing requires a loop.
But yeah thanks a lot it's great to be supported ! I'll help as much as I can too !
First  |  Prev  | 
Pages: 1 2 3 4
Prev: Command to display what Matlab is doing?
Next: Muscle Wrapping over an cylinder - Shortest path