Prev: need help in plotting lines between points
Next: clearing current folders list
From: Levent on 17 Jul 2010 00:17
Dear Matlab Users,

As I understand the Hessian for the fmincon function is somehow calculated in another way than that of the fminunc which makes the standard errors that are calculated by using the Hessian of fmincon not the best possible approximation to the estimated standard errors.

My questions in here are: Is it valid to use Hessian from fmincon? Is this just a numerical issue that might happen or a general problem? How should we calculate the Hessian in this case then? Thanks.

Best,
From: Brian Borchers on 17 Jul 2010 07:41
On Jul 16, 10:17 pm, "Levent " <kutlu...(a)gmail.com> wrote:
> Dear Matlab Users,
>
> As I understand the Hessian for the fmincon function is somehow calculated in another way than that of the fminunc which makes the standard errors that are calculated by using the Hessian of fmincon not the best possible approximation to the estimated standard errors.
>
> My questions in here are: Is it valid to use Hessian from fmincon? Is this just a numerical issue that might happen or a general problem? How should we calculate the Hessian in this case then? Thanks.
>

The context isn't completely clear, so let me start by seeing if I
understand your problem.

You're attempting to estimate some paramters by solving a nonlinear
least squares problem. Right?

In the past, you've used fminunc to minimize the nonlinear least
squares problem, and then used Hessian of the least squares objective
function at the optimal parameter values to get standard errors for
the fitted parameters. Right?

Now, you'd like to repeat the process but with constraints on the
parameters and using fmincon instead of fminunc. Right?

Assuming that I've made all of the right quesses above, then

1. The Hessian returned by fmincon() is the Hessian of the Lagrangian
(incorporating the constraints and the objective function.) This
isn't the same thing as the Hessian of the objective function. In
any case, you can't construct meaningful confidence intervals for the
fitted parameters from the Hessian of the objective function because
of the constraints.

2. Depending on the options to fmincon and fminunc, the Hessian
returned by optimization routine could be a quasi-Newton approximation
to the actual Hessian of the objective function (fminunc) or
Lagrangian (fmincon) In either case, this approximation can be
extremely inaccurate as an approximation to the Hessian- it doesn't
have to be very accurate to be useful in finding search directions
that take the algorithm to an optimal solution.
From: Levent on 17 Jul 2010 13:53
Brian Borchers <borchers.brian(a)gmail.com> wrote in message <5efc15d5-e5aa-40d7-9f16-cbe33b01b487(a)w12g2000yqj.googlegroups.com>...
> On Jul 16, 10:17 pm, "Levent " <kutlu...(a)gmail.com> wrote:
> > Dear Matlab Users,
> >
> > As I understand the Hessian for the fmincon function is somehow calculated in another way than that of the fminunc which makes the standard errors that are calculated by using the Hessian of fmincon not the best possible approximation to the estimated standard errors.
> >
> > My questions in here are: Is it valid to use Hessian from fmincon? Is this just a numerical issue that might happen or a general problem? How should we calculate the Hessian in this case then? Thanks.
> >
>
> The context isn't completely clear, so let me start by seeing if I
> understand your problem.
>
> You're attempting to estimate some paramters by solving a nonlinear
> least squares problem. Right?
>
> In the past, you've used fminunc to minimize the nonlinear least
> squares problem, and then used Hessian of the least squares objective
> function at the optimal parameter values to get standard errors for
> the fitted parameters. Right?
>
> Now, you'd like to repeat the process but with constraints on the
> parameters and using fmincon instead of fminunc. Right?
>
> Assuming that I've made all of the right quesses above, then
>
> 1. The Hessian returned by fmincon() is the Hessian of the Lagrangian
> (incorporating the constraints and the objective function.) This
> isn't the same thing as the Hessian of the objective function. In
> any case, you can't construct meaningful confidence intervals for the
> fitted parameters from the Hessian of the objective function because
> of the constraints.
>
> 2. Depending on the options to fmincon and fminunc, the Hessian
> returned by optimization routine could be a quasi-Newton approximation
> to the actual Hessian of the objective function (fminunc) or
> Lagrangian (fmincon) In either case, this approximation can be
> extremely inaccurate as an approximation to the Hessian- it doesn't
> have to be very accurate to be useful in finding search directions
> that take the algorithm to an optimal solution.


Thank you for Brian. I am talking about a MLE optimization problem. I know that whenever there are constraints the Hessian is not a good approximation for the standard errors. Let me be more precise about about question: Assume that I want to use fmincon to assure that a parameter is non-zero. But I know that this parameter is not zero! This parameter might be standard error parameter which might be negative or positive and I want to assure that the final estimates will pick the positive one. Then in this case the constraints indeed do not bind. Do we still have issues for this case?

I know that it is possible to assure non-negativity by some transformations of parameters. But I still would like to see whether other techniques works or not.

About your point 2): let's assume that we are at the optimal point somehow and the constraints do not bind. In that case I presume that the Hessians are correct (for calculating standard errors)?

Thanks
From: Brian Borchers on 17 Jul 2010 20:04
On Jul 17, 11:53 am, "Levent " <kutlu...(a)gmail.com> wrote:

> Thank you for Brian. I am talking about a MLE optimization problem. I know that whenever there are constraints the Hessian is not a good approximation for the standard errors. Let me be more precise about about question: Assume that I want to use fmincon to assure that a parameter is non-zero. But I know that this parameter is not zero! This parameter might be standard error parameter which might be negative or positive and I want to assure that the final estimates will pick the positive one. Then in this case the constraints indeed do not bind. Do we still have issues for this case?

Yes, if the constraints are all non-binding at optimality, then the
Hessian of the objective function can be used in this way.
Furthermore, if your constraints are all non-binding at optimality,
then the Lagrange multipliers should all be zero, and the Hessian of
the Lagrangian is the Hessian of the objective function.


> About your point 2): let's assume that we are at the optimal point somehow and the constraints do not bind. In that case I presume that the Hessians are correct (for calculating standard errors)?

Here's the problem- if you use a quasi-Newton method to approximate
the Hessian of the Lagrangian for purposes of optimization, that quasi-
Newton approximate Hessian is unlikely to be a good (for purposes of
standard errors) approximation to the actual Hessian, and you should
not use it for calculating standard errors.

Are you supplying the Hessian to fmincon, or are you letting fmincon
do it? If fmincon is approximating the Hessian, then you're using a
quasi-Newton method and you can't safely use the Hessian output by
fmincon for the purpose of computing standard errors.


From: Matt J on 17 Jul 2010 20:06
"Levent " <kutlulev(a)gmail.com> wrote in message <i1sqm0$jnf$1(a)fred.mathworks.com>...

> About your point 2): let's assume that we are at the optimal point somehow and the constraints do not bind. In that case I presume that the Hessians are correct (for calculating standard errors)?
=======

No. Again, depending on various fmincon parameters, the Hessians it computes may be very crude estimates.
 |  Next  |  Last
Pages: 1 2
Prev: need help in plotting lines between points
Next: clearing current folders list