Prev: need help in plotting lines between points
Next: clearing current folders list
From: Levent on 17 Jul 2010 21:00
"Matt J " <mattjacREMOVE(a)THISieee.spam> wrote in message <i1tghc$t20$1(a)fred.mathworks.com>...
> "Levent " wrote in message <i1sqm0$jnf$1(a)fred.mathworks.com>...
>
> > About your point 2): let's assume that we are at the optimal point somehow and the constraints do not bind. In that case I presume that the Hessians are correct (for calculating standard errors)?
> =======
>
> No. Again, depending on various fmincon parameters, the Hessians it computes may be very crude estimates.

Does this imply that we cannot approximate the relevant Hessian matrix in a numerical way? I am pretty sure that it is not always possible to get the analytical Hessiian matrix and we need some kind of approximation. Does calculating the numerical hessian manually by numerical differentiation help? If it does why wouldn't Matlab do it? Thanks.
From: Brian Borchers on 17 Jul 2010 22:04
>
> Does this imply that we cannot approximate the relevant Hessian matrix in a numerical way? I am pretty sure that it is not always possible to get the analytical Hessiian matrix and we need some kind of approximation. Does calculating the numerical hessian manually by numerical differentiation help? If it does why wouldn't Matlab do it? Thanks.

It's not that a numerical approximation to the Hessian cannot be
computed, but rather that it isn't worth computing one for purposes of
optimization. The very crude approximation used in the quasi-Newton
methods is sufficient to get the optimization algorithm to the
optimum. The job of fmincon is optimization, not statistical
parameter estimation- it should be no surprise that the special needs
of statisticians aren't necessarily met by a general purpose
optimization routine.

Depending on the form of your likelihood function (not just its
mathematical form, but also how you compute it), it may be relatively
easy to compute a good approximate Hessian that could reasonably be
used to compute your standard errors. For example, you might be able
to use an automatic differentiation (AD) tool to translate a program
for computing the likehood into a program for computing the likelihood
and its Hessian. Even if this computation is slow, you'd only need to
run it at the optimum solution, not at all of the intermediate
solutions that the quasi-Newton method considered. Or, if your
likelihood was Gaussian, and you were just solving a nonlinear least
squares problem, then it would be straight forward to compute the
Jacobian (matrix of first derivatives) by finite differencing and then
use the approximation H=J'*J. I would recommend against trying to
compute the second derivatives directly by second order finite
differencing- in my experience it just doesn't work well in
practice.

Another approach that might be more practical would be to sample from
your likelihood (or more generally posterior) distribution using
Markov Chain Monte Carlo methods. If the likelihood can be computed
relatively quickly, then this can be a very effective
technique.

A third approach that you might consider is building a quadratic
metamodel of the likelihood near the optimal parameter values.
From: Levent on 17 Jul 2010 22:35
Brian Borchers <borchers.brian(a)gmail.com> wrote in message <1f065a3e-490d-4f5f-9ddc-f9568dad1f7d(a)e5g2000yqn.googlegroups.com>...
> >
> > Does this imply that we cannot approximate the relevant Hessian matrix in a numerical way? I am pretty sure that it is not always possible to get the analytical Hessiian matrix and we need some kind of approximation. Does calculating the numerical hessian manually by numerical differentiation help? If it does why wouldn't Matlab do it? Thanks.
>
> It's not that a numerical approximation to the Hessian cannot be
> computed, but rather that it isn't worth computing one for purposes of
> optimization. The very crude approximation used in the quasi-Newton
> methods is sufficient to get the optimization algorithm to the
> optimum. The job of fmincon is optimization, not statistical
> parameter estimation- it should be no surprise that the special needs
> of statisticians aren't necessarily met by a general purpose
> optimization routine.
>
> Depending on the form of your likelihood function (not just its
> mathematical form, but also how you compute it), it may be relatively
> easy to compute a good approximate Hessian that could reasonably be
> used to compute your standard errors. For example, you might be able
> to use an automatic differentiation (AD) tool to translate a program
> for computing the likehood into a program for computing the likelihood
> and its Hessian. Even if this computation is slow, you'd only need to
> run it at the optimum solution, not at all of the intermediate
> solutions that the quasi-Newton method considered. Or, if your
> likelihood was Gaussian, and you were just solving a nonlinear least
> squares problem, then it would be straight forward to compute the
> Jacobian (matrix of first derivatives) by finite differencing and then
> use the approximation H=J'*J. I would recommend against trying to
> compute the second derivatives directly by second order finite
> differencing- in my experience it just doesn't work well in
> practice.
>
> Another approach that might be more practical would be to sample from
> your likelihood (or more generally posterior) distribution using
> Markov Chain Monte Carlo methods. If the likelihood can be computed
> relatively quickly, then this can be a very effective
> technique.
>
> A third approach that you might consider is building a quadratic
> metamodel of the likelihood near the optimal parameter values.


Thank you. Now, I see why in many of the codes that I see why people are recalculating the Hessian from the scratch rather than using the fminunc or fmincon ones. I thought there should be reason but did not exactly know the reason.

Best,
From: Alan Weiss on 19 Jul 2010 08:36
On 7/17/2010 10:35 PM, Levent wrote:

> Thank you. Now, I see why in many of the codes that I see why people are
> recalculating the Hessian from the scratch rather than using the fminunc
> or fmincon ones. I thought there should be reason but did not exactly
> know the reason.
> Best,

There is at least one function on the File Exchange that will calculate
the Hessian. See
http://www.mathworks.com/matlabcentral/newsreader/view_thread/271183#711741

For more information on the quality of Hessians from various solvers and
algorithms, see
http://www.mathworks.com/access/helpdesk/help/toolbox/optim/ug/bsapd3i.html

Alan Weiss
MATLAB mathematical toolbox documentation
First  |  Prev  | 
Pages: 1 2
Prev: need help in plotting lines between points
Next: clearing current folders list