Mex file to speed up a data remapping problem
in [Matlab]
|
Prev: Vehicle Detection
Next: Optimization of a loop
From: Dan on 4 Aug 2010 17:39 "James Tursa" <aclassyguy_with_a_k_not_a_c(a)hotmail.com> wrote in message <i3cled$9o0$1(a)fred.mathworks.com>... > "Dan " <danfuppd(a)gmaill.com> wrote in message <i3cgd4$iv8$1(a)fred.mathworks.com>... > > > > However, one thing that causes the problem for me now is that the size of array is very large, if everytime B and I are passed into and then out of the mex file, it will do many numbers of array copies which slows down the whole process. Is there any where to avoid that and somehow treat the arrays as a global variable and only pointers are needed to operate on the elements of the array? > > Arguments passed to mex files are passed by reference ... there are no copies made. > > James Tursa Thanks, James. This is somewhat different from what I read. Do you mean that I can operate on the large array B and I within the mex file and return it back to the output without memory usage increase due to array copying? The what should I assign to plhs? Possibly something like this: B = mxGetPr(prhs[0]); I = mxGetPr(prhs[1]); / operation on array B and I involving elements change / mxSetPr(plhs[0], B); mxSetPr(plhs[1],I); Dan
From: Dan on 4 Aug 2010 17:48 Walter Roberson <roberson(a)hushmail.com> wrote in message > Correction: earlier where I wrote @avg I should have written @mean > > mod1 = mod(I, 1); > Bsum = accumarray([I;I+1], [A.*(1-mod1); A.*mod1]); > > However, you would now have to figure out what kind of averaging you want to > use. Is it fair to use a full extra count of 1 for an instance that is only > contributing 0.2 of a sample ? If it is fair, then why would it not also be > fair to use a full extra repeat count of 1 for an instance that is > contributing eps() of a simple, or contributing 0 of a sample? > > I deliberately omitted the @mean call in the above (letting it default to > @sum) because an index I(K) that happens to be exactly an integer would > contribute a full sample to the location I(K) and would also contribute a 0 to > the location I(K)+1 and I suspect that you do not want that 0 to count towards > the repeat count in that case... but if you don't, then what if the index > value is not _exactly_ an integer, and so on. > > If your A values cannot be 0 naturally, then you could define > > meanNonZero = @(V) mean(V(V~=0)); > > and use meanNonZero as the function to supply to accumarray . If you do that, > though, watch out as you could end up with some NaN values. This would work out very nicely if the data is complete. However, in my case, array A have a lot of elements with value NaN. Therefore accumarray does not work. Another complicating situation is that the actual array A in the experiment is a 3D matrix. So we are remapping 3D dataset (A, say 512*512*3000, with nonuniform index) to 3D griddata (B, 256*256*256, uniform index). Linear interpolation requires duplicating the matrix 8 times even if the above method can be used. Due to the complexity of the data and need for interpolation, I would like to perform for loop that can check the data (NaN or not), remap to grid point and also do the interpolation if possible. With Matlab, it is simply too slow for our purpose.
From: Walter Roberson on 4 Aug 2010 18:21 Dan wrote: > This would work out very nicely if the data is complete. However, in my > case, array A have a lot of elements with value NaN. Therefore > accumarray does not work. meannonnan = @(V) mean(V(~isnan(V))); > Another complicating situation is that the > actual array A in the experiment is a 3D matrix. So we are remapping 3D > dataset (A, say 512*512*3000, with nonuniform index) to 3D griddata (B, > 256*256*256, uniform index). Possibly John D'Errico's FEX contributions "gridfit" or "consolidator" might be of benefit. And possibly his "inpaint_nans" as well. > Linear interpolation requires duplicating > the matrix 8 times even if the above method can be used. You are moving the goal-posts :( Consider A(end,1,1) and A(1,2,1). They have adjacent linear indices -- in the case of your example sizes, linear indices 512 and 513 respectively. What would it mean if the fractional index I(K) was between 512 and 513? Which points would you want to interpolate from, and with what weightings? If the interpolation is to be drawn from A, then what if some of those points in A happen to be NaN, then how do you want the interpolation to proceed? > Due to the > complexity of the data and need for interpolation, I would like to > perform for loop that can check the data (NaN or not), remap to grid > point and also do the interpolation if possible. With Matlab, it is simply > too slow for our purpose. If we do not have a rigorous description of the computation (and the memory constraints) then we have little chance of proposing efficient code.
From: James Tursa on 4 Aug 2010 19:42 "Dan " <danfuppd(a)gmaill.com> wrote in message <i3cmlo$q65$1(a)fred.mathworks.com>... > "James Tursa" <aclassyguy_with_a_k_not_a_c(a)hotmail.com> wrote in message <i3cled$9o0$1(a)fred.mathworks.com>... > > "Dan " <danfuppd(a)gmaill.com> wrote in message <i3cgd4$iv8$1(a)fred.mathworks.com>... > > > > > > However, one thing that causes the problem for me now is that the size of array is very large, if everytime B and I are passed into and then out of the mex file, it will do many numbers of array copies which slows down the whole process. Is there any where to avoid that and somehow treat the arrays as a global variable and only pointers are needed to operate on the elements of the array? > > > > Arguments passed to mex files are passed by reference ... there are no copies made. > > > > James Tursa > Thanks, James. This is somewhat different from what I read. Do you mean that I can operate on the large array B and I within the mex file and return it back to the output without memory usage increase due to array copying? The what should I assign to plhs? Possibly something like this: > > B = mxGetPr(prhs[0]); > I = mxGetPr(prhs[1]); > / operation on array B and I involving elements change / > mxSetPr(plhs[0], B); > mxSetPr(plhs[1],I); MATLAB does not officially support or endorse any in-place operations in mex routines. That being said, if you know what you are doing then you can indeed do it and avoid data copies. However, if the variable is being shared with another variable in the MATLAB workspace, then you will change *both* variables ... that is why you have to know what you are doing so you can avoid this undesirable side effect. For your code snippet above, you would not use mxSetPr. You would just do this: B = mxGetPr(prhs[0]); I = mxGetPr(prhs[1]); / operation on array B and I involving elements change / Then you are done. Since B and I are pointing *directly* at the data area of the original variable in the MATLAB workspace, you will immediately change the contents of the MATLAB workspace variable when you do this. There is nothing to set as far as the plhs array is concerned. i.e., don't do *anything* with the plhs array. There is no need to set any plhs since you are not returning anything back to MATLAB as far as a function output is concerned ... you are simply modifying the data contents of an existing variable in-place. In fact, what I would do is generate an error (call mexErrMsgTxt) if nlhs > 0 in this case. James Tursa
From: Dan on 6 Aug 2010 09:12
Thanks, Walter. Your technique works if there is no interpolation. I am sorry about moving the goal post because I think it is easier to describe the problem without putting in too much details at once. The actual interpolation needs to be done in 3D matrix and therefore it is not easy to describe the problem in a few line. Anyway, I have tried mex file and it speeds up the calculation more than 10 times. For the time being, I will stick with that. |