Prev: Help! Cannot start MATLAB in Fedora 8
Next: Rayleighchan - Independent Instantiations
From: Quynh Nga Nghiem on 4 May 2008 18:16
Hello,

I'm having problems with using "fmincon" to minimize an
objective function that is a negative log likelihood function.
Inside the objective function, I already have to use
"fsolve" as a nested function to find the value of an input
for my objective function to be used in "fmincon".

It seems that both fmincon and fsolve are very dependent to
the initial values I put in. For fsolve, once fmincon change
the parameters for its next step, fsolve requires me another
starting point. Other wise it will report:

"Optimizer appears to be converging to a minimum that is not
a root: Sum of squares of the function values is >
sqrt(options.TolFun). Try again with a new starting point."

But I can only give it 1 starting point from the beginning.
Sometimes, I was lucky to have a good starting point for
fsolve and able to continue fmincon after that, I faced
another problem. Out of 3 parameters I need to estimate,
fmincon only changed 1 parameter and kept the other 2
parameters as the starting point or lower bound/upper bound.
The message I received are below:

"Optimization terminated: magnitude of directional
derivative in search direction less than 2*options.TolFun
and maximum constraint violation is less than options.TolCon.
Active inequalities (to within options.TolCon = 1e-006):
lower upper ineqlin ineqnonlin
2

I struggled with the problem for weeks but can not
understand how to deal with it. I would be very grateful if
any of you could help me as it is very very important for me
at the moment. (I'm using Matlab version 7.0.1)

Thank you so much for reading this message and give me some
ideas about it.
From: helper on 5 May 2008 04:56
"Quynh Nga Nghiem" <nghiemquynhnga(a)yahoo.com> wrote in
message <fvlcj3$9v0$1(a)fred.mathworks.com>...
> Hello,
>
> I'm having problems with using "fmincon" to minimize an
> objective function that is a negative log likelihood
function.
> Inside the objective function, I already have to use
> "fsolve" as a nested function to find the value of an
input
> for my objective function to be used in "fmincon".
>
> It seems that both fmincon and fsolve are very dependent
to
> the initial values I put in. For fsolve, once fmincon
change
> the parameters for its next step, fsolve requires me
another
> starting point. Other wise it will report:
>
> "Optimizer appears to be converging to a minimum that is
not
> a root: Sum of squares of the function values is >
> sqrt(options.TolFun). Try again with a new starting
point."
>
> But I can only give it 1 starting point from the
beginning.
> Sometimes, I was lucky to have a good starting point for
> fsolve and able to continue fmincon after that, I faced
> another problem. Out of 3 parameters I need to estimate,
> fmincon only changed 1 parameter and kept the other 2
> parameters as the starting point or lower bound/upper
bound.
> The message I received are below:
>
> "Optimization terminated: magnitude of directional
> derivative in search direction less than 2*options.TolFun
> and maximum constraint violation is less than
options.TolCon.
> Active inequalities (to within options.TolCon = 1e-006):
> lower upper ineqlin ineqnonlin
> 2
>
> I struggled with the problem for weeks but can not
> understand how to deal with it. I would be very grateful
if
> any of you could help me as it is very very important for
me
> at the moment. (I'm using Matlab version 7.0.1)
>
> Thank you so much for reading this message and give me
some
> ideas about it.


Until someone more knowledgeable responds, I can give a few
tips.

First, when using any optimization function, you usually
need to be very knowledgeable about what your objective
functions look like. If you have no knowledge about what
is a good first guess for your FSOLVE function based on the
variables being altered by FMINCON, you are probably not
going to be able to use this method.

Investigate the nature of your FSOLVE objective over a
range of variables and see if you can determine some useful
heuristic for determining a good first guess.


Second, FMINCON expects your objective function to be
smooth (twice continuously differentiable). I suspect that
your use of FSOLVE within the objective for FMINCON is
going to result in an FMINCON object that is not smooth.

Third, rather than using FSOLVE, why not specify the FSOLVE
objective as a (possibly nonlinear) constraint within
FMINCON (if this is possible for your problem)?


Fourth, look into using the PATTERNSEARCH function within
the Genetic Algorithm and Direct Search Toolbox (if you
don't have it, get a trial version). This function is much
less susceptible to poor initial guesses.

From: John D'Errico on 5 May 2008 07:09
"Quynh Nga Nghiem" <nghiemquynhnga(a)yahoo.com> wrote in message
<fvlcj3$9v0$1(a)fred.mathworks.com>...
> Hello,
>
> I'm having problems with using "fmincon" to minimize an
> objective function that is a negative log likelihood function.
> Inside the objective function, I already have to use
> "fsolve" as a nested function to find the value of an input
> for my objective function to be used in "fmincon".

This is a recipe for danger. The presumption
that almost any optimizer starts with is that
the objective function is continuous and
differentiable.

When your objective is not so, expect to see
problems. Why is an objective that itself calls
fsolve not differentiable? You only converge
to within some tolerance. But then the fmincon
call will try to differentiate that, getting a
gradient vector. How do you form a numerical
derivative? You subtract your function value at
two points that are VERY close to each other,
then divide by the difference in x.

If you have an fsolve call inside the fmincon
objective, then that approximation to the
derivative using a finite difference will result
in essentially random trash.

Better is to use an optimizer on the outside
that is less susceptible to subtle trash like
this. Fminsearch is one, or perhaps a genetic
algorithm or other stochastic optimizer, like
particle swarms or simulated annealing.

These alternatives are better choices because
they never try to form a gradient vector.


> It seems that both fmincon and fsolve are very dependent to
> the initial values I put in. For fsolve, once fmincon change
> the parameters for its next step, fsolve requires me another
> starting point. Other wise it will report:

Yes, all optimizers are sensitive to their starting
values. The comparison I like to give is a simple
task. Give a blind person the job of finding the
lowest spot on earth. Put them down in some
random, arbitrary location on the surface, and
give them only an altimeter, which will read out
their altitude (or depth) at any location. (Be nice
and give your subject scuba gear to wear.)

What are the odds, if you start this fellow out in
the midst of the Himalayas, that they will find
the depths of an ocean trench in the Pacific? Or
perhaps, there is some bore hole somewhere
that goes deeper? How would they possibly ever
find that spot?

Any local solution to an optimization problem
has what is called a basin of attraction. This is
the set of points which when used as starting
values for a given optimizer, will end up at a
given local minimizer. I discuss these things in
my optimization tips and tricks document on the
file exchange.


> "Optimizer appears to be converging to a minimum that is not
> a root: Sum of squares of the function values is >
> sqrt(options.TolFun). Try again with a new starting point."
>
> But I can only give it 1 starting point from the beginning.
> Sometimes, I was lucky to have a good starting point for
> fsolve and able to continue fmincon after that, I faced
> another problem. Out of 3 parameters I need to estimate,
> fmincon only changed 1 parameter and kept the other 2
> parameters as the starting point or lower bound/upper bound.
> The message I received are below:
>
> "Optimization terminated: magnitude of directional
> derivative in search direction less than 2*options.TolFun
> and maximum constraint violation is less than options.TolCon.
> Active inequalities (to within options.TolCon = 1e-006):
> lower upper ineqlin ineqnonlin
> 2

(Re)read my other comments. This is a not
unsurprising result given what you are trying.


> I struggled with the problem for weeks but can not
> understand how to deal with it. I would be very grateful if
> any of you could help me as it is very very important for me
> at the moment. (I'm using Matlab version 7.0.1)
>
> Thank you so much for reading this message and give me some
> ideas about it.

I'm sorry. But there are no magic solutions
that I can be offer.

John


From: John D'Errico on 5 May 2008 07:09
"Quynh Nga Nghiem" <nghiemquynhnga(a)yahoo.com> wrote in message
<fvlcj3$9v0$1(a)fred.mathworks.com>...
> Hello,
>
> I'm having problems with using "fmincon" to minimize an
> objective function that is a negative log likelihood function.
> Inside the objective function, I already have to use
> "fsolve" as a nested function to find the value of an input
> for my objective function to be used in "fmincon".

This is a recipe for danger. The presumption
that almost any optimizer starts with is that
the objective function is continuous and
differentiable.

When your objective is not so, expect to see
problems. Why is an objective that itself calls
fsolve not differentiable? You only converge
to within some tolerance. But then the fmincon
call will try to differentiate that, getting a
gradient vector. How do you form a numerical
derivative? You subtract your function value at
two points that are VERY close to each other,
then divide by the difference in x.

If you have an fsolve call inside the fmincon
objective, then that approximation to the
derivative using a finite difference will result
in essentially random trash.

Better is to use an optimizer on the outside
that is less susceptible to subtle trash like
this. Fminsearch is one, or perhaps a genetic
algorithm or other stochastic optimizer, like
particle swarms or simulated annealing.

These alternatives are better choices because
they never try to form a gradient vector.


> It seems that both fmincon and fsolve are very dependent to
> the initial values I put in. For fsolve, once fmincon change
> the parameters for its next step, fsolve requires me another
> starting point. Other wise it will report:

Yes, all optimizers are sensitive to their starting
values. The comparison I like to give is a simple
task. Give a blind person the job of finding the
lowest spot on earth. Put them down in some
random, arbitrary location on the surface, and
give them only an altimeter, which will read out
their altitude (or depth) at any location. (Be nice
and give your subject scuba gear to wear.)

What are the odds, if you start this fellow out in
the midst of the Himalayas, that they will find
the depths of an ocean trench in the Pacific? Or
perhaps, there is some bore hole somewhere
that goes deeper? How would they possibly ever
find that spot?

Any local solution to an optimization problem
has what is called a basin of attraction. This is
the set of points which when used as starting
values for a given optimizer, will end up at a
given local minimizer. I discuss these things in
my optimization tips and tricks document on the
file exchange.


> "Optimizer appears to be converging to a minimum that is not
> a root: Sum of squares of the function values is >
> sqrt(options.TolFun). Try again with a new starting point."
>
> But I can only give it 1 starting point from the beginning.
> Sometimes, I was lucky to have a good starting point for
> fsolve and able to continue fmincon after that, I faced
> another problem. Out of 3 parameters I need to estimate,
> fmincon only changed 1 parameter and kept the other 2
> parameters as the starting point or lower bound/upper bound.
> The message I received are below:
>
> "Optimization terminated: magnitude of directional
> derivative in search direction less than 2*options.TolFun
> and maximum constraint violation is less than options.TolCon.
> Active inequalities (to within options.TolCon = 1e-006):
> lower upper ineqlin ineqnonlin
> 2

(Re)read my other comments. This is a not
unsurprising result given what you are trying.


> I struggled with the problem for weeks but can not
> understand how to deal with it. I would be very grateful if
> any of you could help me as it is very very important for me
> at the moment. (I'm using Matlab version 7.0.1)
>
> Thank you so much for reading this message and give me some
> ideas about it.

I'm sorry. But there are no magic solutions
that I can be offer.

John


From: John D'Errico on 5 May 2008 07:11
"Quynh Nga Nghiem" <nghiemquynhnga(a)yahoo.com> wrote in message
<fvlcj3$9v0$1(a)fred.mathworks.com>...
> Hello,
>
> I'm having problems with using "fmincon" to minimize an
> objective function that is a negative log likelihood function.
> Inside the objective function, I already have to use
> "fsolve" as a nested function to find the value of an input
> for my objective function to be used in "fmincon".

This is a recipe for danger. The presumption
that almost any optimizer starts with is that
the objective function is continuous and
differentiable.

When your objective is not so, expect to see
problems. Why is an objective that itself calls
fsolve not differentiable? You only converge
to within some tolerance. But then the fmincon
call will try to differentiate that, getting a
gradient vector. How do you form a numerical
derivative? You subtract your function value at
two points that are VERY close to each other,
then divide by the difference in x.

If you have an fsolve call inside the fmincon
objective, then that approximation to the
derivative using a finite difference will result
in essentially random trash.

Better is to use an optimizer on the outside
that is less susceptible to subtle trash like
this. Fminsearch is one, or perhaps a genetic
algorithm or other stochastic optimizer, like
particle swarms or simulated annealing.

These alternatives are better choices because
they never try to form a gradient vector.


> It seems that both fmincon and fsolve are very dependent to
> the initial values I put in. For fsolve, once fmincon change
> the parameters for its next step, fsolve requires me another
> starting point. Other wise it will report:

Yes, all optimizers are sensitive to their starting
values. The comparison I like to give is a simple
task. Give a blind person the job of finding the
lowest spot on earth. Put them down in some
random, arbitrary location on the surface, and
give them only an altimeter, which will read out
their altitude (or depth) at any location. (Be nice
and give your subject scuba gear to wear.)

What are the odds, if you start this fellow out in
the midst of the Himalayas, that they will find
the depths of an ocean trench in the Pacific? Or
perhaps, there is some bore hole somewhere
that goes deeper? How would they possibly ever
find that spot?

Any local solution to an optimization problem
has what is called a basin of attraction. This is
the set of points which when used as starting
values for a given optimizer, will end up at a
given local minimizer. I discuss these things in
my optimization tips and tricks document on the
file exchange.


> "Optimizer appears to be converging to a minimum that is not
> a root: Sum of squares of the function values is >
> sqrt(options.TolFun). Try again with a new starting point."
>
> But I can only give it 1 starting point from the beginning.
> Sometimes, I was lucky to have a good starting point for
> fsolve and able to continue fmincon after that, I faced
> another problem. Out of 3 parameters I need to estimate,
> fmincon only changed 1 parameter and kept the other 2
> parameters as the starting point or lower bound/upper bound.
> The message I received are below:
>
> "Optimization terminated: magnitude of directional
> derivative in search direction less than 2*options.TolFun
> and maximum constraint violation is less than options.TolCon.
> Active inequalities (to within options.TolCon = 1e-006):
> lower upper ineqlin ineqnonlin
> 2

(Re)read my other comments. This is a not
unsurprising result given what you are trying.


> I struggled with the problem for weeks but can not
> understand how to deal with it. I would be very grateful if
> any of you could help me as it is very very important for me
> at the moment. (I'm using Matlab version 7.0.1)
>
> Thank you so much for reading this message and give me some
> ideas about it.

I'm sorry. But there are no magic solutions
that I can be offer.

John


 |  Next  |  Last
Pages: 1 2
Prev: Help! Cannot start MATLAB in Fedora 8
Next: Rayleighchan - Independent Instantiations