Prev: query on matlab
Next: Can you stop and continue execution, or check the progress of the
From: Wojtek Kurek on 26 Jul 2010 08:48
Hello,

I've been struggling for few past days trying to run a batch job (from
parallel computing toolbox) containing a mex file. The mex file for me
is a way of launching an external simulation using Epanet (water
network simulator) all works well if I use it form a script however if
I run it like this

matlabpool
test = batch('script.m')
wait(test)
load(test)
.....


I get a matlab error on loading results and the script when launched
locally run script... has no problems at all to complete.

I tried using 'FileDependencies' property for the batch command and
added all the extra mex file resources (i.e. used dll library
containing epanet... ) with no success.

From: Rune Allnor on 26 Jul 2010 09:07
On 26 Jul, 14:48, Wojtek Kurek <woj.ku...(a)gmail.com> wrote:
> Hello,
>
> I've been struggling for few past days trying to run a batch job (from
> parallel computing toolbox) containing a mex file. The mex file for me
> is a way of launching an external simulation using Epanet (water
> network simulator)

So if I understand this correctly, you

1) Call a matlab batch job to run a script
2) The script calls a MEX file
3) The MEX file invokes some external program
4) Matlab complains about error on loading the
end result from file?

If my understanding is correct, this is likely a
question about processes being out of sync. Don't be
surprised if the external program in step 3 is
invoked as a different process than matlab, in which
case there is no way matlab can tell how long time
the external program takes to finish. I'd suspect that
matlab / mex starts the external program, and then
immediately returns to try and load the output files
that the external program has not yet produced.

I would suggest you skip the MEX file and instead
call the exetrnal program directly from matlab through
a SYSTEM call. That way matlab has a fair chance of
keeping track of when the external program has finished.

Rune
From: Wojtek Kurek on 26 Jul 2010 09:22

>
> So if I understand this correctly, you
>
> 1) Call a matlab batch job to run a script
> 2) The script calls a MEX file
> 3) The MEX file invokes some external program
> 4) Matlab complains about error on loading the
>    end result from file?
>
> If my understanding is correct, this is likely a
> question about processes being out of sync. Don't be
> surprised if the external program in step 3 is
> invoked as a different process than matlab, in which
> case there is no way matlab can tell how long time
> the external program takes to finish. I'd suspect that
> matlab / mex starts the external program, and then
> immediately returns to try and load the output files
> that the external program has not yet produced.
>
> I would suggest you skip the MEX file and instead
> call the exetrnal program directly from matlab through
> a SYSTEM call. That way matlab has a fair chance of
> keeping track of when the external program has finished.
>
> Rune

You understood it quite right except for one thing, the external
program is not a standalone application its only a dll functions
library, and those functions are invoked by a Matlab MEX file and
Matlab should be able to monitor the execution of MEX files... maybe I
need to specify some more config parameters for the batch command...

The main goal of my work now is to parallelize my GA and I wanted to
evaluate objective functions in parallel... but from what I see now it
may be quite difficult...

Wojtek
From: Rune Allnor on 26 Jul 2010 09:33
On 26 Jul, 15:22, Wojtek Kurek <woj.ku...(a)gmail.com> wrote:
> > So if I understand this correctly, you
>
> > 1) Call a matlab batch job to run a script
> > 2) The script calls a MEX file
> > 3) The MEX file invokes some external program
> > 4) Matlab complains about error on loading the
> >    end result from file?
>
> > If my understanding is correct, this is likely a
> > question about processes being out of sync. Don't be
> > surprised if the external program in step 3 is
> > invoked as a different process than matlab, in which
> > case there is no way matlab can tell how long time
> > the external program takes to finish. I'd suspect that
> > matlab / mex starts the external program, and then
> > immediately returns to try and load the output files
> > that the external program has not yet produced.
>
> > I would suggest you skip the MEX file and instead
> > call the exetrnal program directly from matlab through
> > a SYSTEM call. That way matlab has a fair chance of
> > keeping track of when the external program has finished.
>
> > Rune
>
> You understood it quite right except for one thing, the external
> program is not a standalone application its only a dll functions
> library, and those functions are invoked by a Matlab MEX file

OK. It needs not help much, as the same problem might still apply
if the writing of the result files from the DLL is delegated to
the OS as stand-alone processes. Might very well be the case
in a multi-thread configuration, as the parallel scenario
necessarily is.

One way to achieve your goal is to skip all ideas about parallel
processing at the matlab level, and instead call a number of
stand-alone versions of your DLL functions. That way it will
be up to the OS, not matlab, to distribute the workload out
to the availbale cores.

You would need to ensure that each instance of the stand-alone
process is able to work with individually named parameter files,
and also find a way to have matlab spawn stand-alone processes.

Rune
From: Wojtek Kurek on 26 Jul 2010 09:41
On 26 Lip, 15:33, Rune Allnor <all...(a)tele.ntnu.no> wrote:
> On 26 Jul, 15:22, Wojtek Kurek <woj.ku...(a)gmail.com> wrote:
>
>
>
>
>
> > > So if I understand this correctly, you
>
> > > 1) Call a matlab batch job to run a script
> > > 2) The script calls a MEX file
> > > 3) The MEX file invokes some external program
> > > 4) Matlab complains about error on loading the
> > >    end result from file?
>
> > > If my understanding is correct, this is likely a
> > > question about processes being out of sync. Don't be
> > > surprised if the external program in step 3 is
> > > invoked as a different process than matlab, in which
> > > case there is no way matlab can tell how long time
> > > the external program takes to finish. I'd suspect that
> > > matlab / mex starts the external program, and then
> > > immediately returns to try and load the output files
> > > that the external program has not yet produced.
>
> > > I would suggest you skip the MEX file and instead
> > > call the exetrnal program directly from matlab through
> > > a SYSTEM call. That way matlab has a fair chance of
> > > keeping track of when the external program has finished.
>
> > > Rune
>
> > You understood it quite right except for one thing, the external
> > program is not a standalone application its only a dll functions
> > library, and those functions are invoked by a Matlab MEX file
>
> OK. It needs not help much, as the same problem might still apply
> if the writing of the result files from the DLL is delegated to
> the OS as stand-alone processes. Might very well be the case
> in a multi-thread configuration, as the parallel scenario
> necessarily is.
>
> One way to achieve your goal is to skip all ideas about parallel
> processing at the matlab level, and instead call a number of
> stand-alone versions of your DLL functions. That way it will
> be up to the OS, not matlab, to distribute the workload out
> to the availbale cores.
>
> You would need to ensure that each instance of the stand-alone
> process is able to work with individually named parameter files,
> and also find a way to have matlab spawn stand-alone processes.
>
> Rune

Well I wanted to avoid this, we have developed a homemade grid
software that used to launch different matlab instances but it would
seem that I will have to remove dust from it...

Thx Rune.
 | 
Pages: 1
Prev: query on matlab
Next: Can you stop and continue execution, or check the progress of the