Prev: streaming in D2Q7 LB model
Next: shape detection
From: ibisek on 6 Oct 2009 06:23
Hello,

I need to access a shared matrix/array/vector/variable from multiple workers/labs and have the following problems:

x=zeros(10,1)
parfor i=1:length(x)
if x(arbitraryIndex) == whatever
x(i) = somecode....
end
end

results in SLICING the 'x' vector and I cannot access all elements.

In:

spmd
myIndexes = indexes(labindex,:);
for i=myIndexes(1):myIndexes(2)
% do whatever you desire with the array 'x'
end
end

I get COPIES of x for every worker. In the final, the array 'x' is not the original matrix but an array of cells of size (num-cpu-cores (=workers)) which differs in the fields which were modified within body of a particular worker.

That means I cannot access the changes concurrently made by other workers. I need a REALLY SHARED array to be accessed at the same time from multiple workers (having in mind collisions may occur here, but that can be included into the plan and could be avoided by workers writing only into a separated segments of the array, however reading from the entire space).

Is there a way?

Thanks a lot for any tips!

Ibisek
From: Bruno Luong on 6 Oct 2009 06:50
I'm not familiar with the parallel toolbox, but I might suggest to take a look at this tool on FEX that might be of your interests.

http://www.mathworks.com/matlabcentral/fileexchange/24576

Bruno
From: Sebastiaan on 6 Oct 2009 07:43
"Bruno Luong" <b.luong(a)fogale.findmycountry> wrote in message <haf7db$avj$1(a)fred.mathworks.com>...
> I'm not familiar with the parallel toolbox, but I might suggest to take a look at this tool on FEX that might be of your interests.
>
> http://www.mathworks.com/matlabcentral/fileexchange/24576
>
> Bruno
Wow, very nice idea. When I experimented with the parallel toolbox a while ago, I hit this problem:

A = HugeStructureWithReadOnlyData;
parfor i=1:n
x(i) = oper(A, i);
end

It resulted copying A to all workers (despite that it is read-only), leading to a serious memory problem.

What were your problems with structures?

Sebastiaan
From: Bruno Luong on 6 Oct 2009 07:59
"Sebastiaan " <s.breedveld(a)erasmusmc.REMOVE.BOO.BOO.nl> wrote in message <hafag6$okj$1(a)fred.mathworks.com>...
> "Bruno Luong" <b.luong(a)fogale.findmycountry> wrote in message <haf7db$avj$1(a)fred.mathworks.com>...
>
> What were your problems with structures?
>

Making similar function for structures and cells demand a serious investigation and reversed engineering on Matlab internal storing. I have spend a small amount of time, but did not get into the bottom.

Peter Boettcher has wrote a long post ago about such internal storing, unfortunately what he wrote is no longer valid for recent Matlab.

Bruno
From: Edric M Ellis on 6 Oct 2009 08:42
"ibisek " <cervenka(a)fai.utb.cz> writes:

> I need to access a shared matrix/array/vector/variable from multiple
> workers/labs and have the following problems:
>
> [ ... parfor ... ]
>
> results in SLICING the 'x' vector and I cannot access all elements.
>
> In:
>
> spmd
> myIndexes = indexes(labindex,:);
> for i=myIndexes(1):myIndexes(2)
> % do whatever you desire with the array 'x'
> end
> end
>
> I get COPIES of x for every worker. In the final, the array 'x' is not the
>original matrix but an array of cells of size (num-cpu-cores (=workers)) which
>differs in the fields which were modified within body of a particular worker.

You need a (co)distributed array, where each worker stores just a portion of the
array. You can then work with the local parts on each worker. Here's an example
using R2009b syntax (note that there are some differences with

spmd
x = codistributed.zeros( 100 ); % 100x100 array spread across the workers
lp = getLocalPart( x ); % The underlying "local part"
rr = globalIndices( x, 1 ); % Which rows in the global array do we have?
cc = globalIndices( x, 2 ); % Which columns?
for ii = 1:length( rr )
for jj = 1:length( cc )
% perform some calculation on the local part
lp(ii, jj) = labindex * (rr(ii) + cc(jj));
end
end
% We've modified the local part of "x", we need to put it back together
x = codistributed.build( lp, getCodistributor( x ) );
end

% Back at the client, x is now a "distributed" array, with the data still on the
% workers. Use "gather" to bring it back to the client
xd = gather( x );

I must say that working directly with the local parts of codistributed arrays in
this way requires some care. In particular, you need to take care to call
"globalIndices" to find out which indices into the global array you have on each
worker, and when you've finished modifying the local part, you must use
"codistributed.build" to put things back together again. If you stick to the
methods defined for (co)distributed arrays - such as mtimes and so on - then the
communication is done for you "under the hood". Otherwise, there are labSend and
friends.

If you're not using R2009b, there are ways of doing all this stuff, but the
syntax is slightly different.

Cheers,

Edric.
 |  Next  |  Last
Pages: 1 2 3
Prev: streaming in D2Q7 LB model
Next: shape detection