Model fit for ordinal multinomial models in GENMOD?
in [SAS]
|
From: Robin R High on 12 Feb 2010 13:50 Tobias, It is somewhat of a multi-step process, namely first determine if the cumulative logit model assumption is OK with LOGISTIC: PROC LOGISTIC DATA=indat; CLASS xvar / PARAM=glm; FREQ count; MODEL y = xvar / link=clogit ; run; Near the top you'll see: The LOGISTIC Procedure Score Test for the Proportional Odds Assumption Chi-Square DF Pr > ChiSq 3.9106 3 0.2713 which the relatively "large" pvalue indicates the cumulative logit model works To assess goodness of fit for the clogit model and a categorical explanatory variable, you can run GENMOD with the aggregate= option PROC genmod DATA=indat; CLASS xvar ; FREQ count; MODEL y = xvar / dist=multinomial link=clogit aggregate=(xvar) ; RUN; that produces the Criteria For Assessing Goodness Of Fit Criterion DF Value Value/DF Deviance 3 3.6877 1.2292 Pearson Chi-Square 3 3.6630 1.2210 which are then compared to a chi-square distribution with the indicated degrees of freedom (DF). One can verify these fit statistics by computing the expected counts for each cell of the table (takes a few programming steps) and connect them with the observed counts entering them into the chisq and dev formulas: expected = total * prob ; where prob is the estimated probability for each response level for each level of xvar total is the sum of all observed counts for each level of the xvar sum these values across all cells: chisq = ( count - expected)**2/expected; dev = 2*count*(log( count / expected)); Note that if the clogit doesn't work (bad fit or proportional odds not satisfied) then enter link=glogit (in LOGISTIC or GLIMMIX)' the expected and observed counts are the same for a categorical xvar Robin High UNMC From: Tobias Jeppsson <tobias.jeppsson(a)EKOL.SLU.SE> To: SAS-L(a)LISTSERV.UGA.EDU Date: 02/11/2010 05:49 AM Subject: Model fit for ordinal multinomial models in GENMOD? Sent by: "SAS(r) Discussion" <SAS-L(a)LISTSERV.UGA.EDU> Hi I have been fitting models for two response variables, one ordinal and one normal (using AIC to choose the "best" model). For the normal data I can evaluate model fit and assumptions as usual, and by looking at residual plots. However, as I understand it, residuals are not available for ordinal multinomial models in GENMOD. How do I then best assess fit and if I meet the model assumptions? This is the first time that I'm analyzing ordinal multinomial models, so maybe I'm missing something obvious. thanks Tobias Jeppsson
From: Tobias Jeppsson on 17 Feb 2010 12:18 Thank you both Robin & Peter for your helpful responses. When looking at the Score Test for the Proportional Odds Assumption (as Robin suggested) it is indeed significant, indicating a violation of the assumptions. However, when trying Peter's suggestion of comparing the nominal and ordinal formulations I find a relatively good correspondence between predictions, when plotting the predictions for all response classes (R2=0.93). To test the "robustness" of the results I also ran the same model assuming dist=poisson (in genmod), even though the assumption of an ordinal response feels more natural. The response categories are (0, 1, 2, 3), with a decreasing number of observation in each category going from 0-3. This analysis come to the same basic conclusion about the effects of my predictors (direction and significance). So basically I feel quite confident in the overall results/patterns in my data, but uneasy about violating the basic assumptions in the ordinal model. What do you think is the best way to proceed? I'm not certain on how to use the AGGREGATE option to assess fit since I have several predictor variables, both class and continuous. Should I assess fit sequentially by looking at the predictor variables one by one? And just to clarify, my response variable is threat classifications of species, with most species considered non-threatened and less species in the higher threat categories. Thanks again Tobias -----Original Message----- From: SAS(r) Discussion [mailto:SAS-L(a)LISTSERV.UGA.EDU] On Behalf Of Robin R High Sent: den 12 februari 2010 19:51 To: SAS-L(a)LISTSERV.UGA.EDU Subject: Re: Model fit for ordinal multinomial models in GENMOD? Tobias, It is somewhat of a multi-step process, namely first determine if the cumulative logit model assumption is OK with LOGISTIC: PROC LOGISTIC DATA=indat; CLASS xvar / PARAM=glm; FREQ count; MODEL y = xvar / link=clogit ; run; Near the top you'll see: The LOGISTIC Procedure Score Test for the Proportional Odds Assumption Chi-Square DF Pr > ChiSq 3.9106 3 0.2713 which the relatively "large" pvalue indicates the cumulative logit model works To assess goodness of fit for the clogit model and a categorical explanatory variable, you can run GENMOD with the aggregate= option PROC genmod DATA=indat; CLASS xvar ; FREQ count; MODEL y = xvar / dist=multinomial link=clogit aggregate=(xvar) ; RUN; that produces the Criteria For Assessing Goodness Of Fit Criterion DF Value Value/DF Deviance 3 3.6877 1.2292 Pearson Chi-Square 3 3.6630 1.2210 which are then compared to a chi-square distribution with the indicated degrees of freedom (DF). One can verify these fit statistics by computing the expected counts for each cell of the table (takes a few programming steps) and connect them with the observed counts entering them into the chisq and dev formulas: expected = total * prob ; where prob is the estimated probability for each response level for each level of xvar total is the sum of all observed counts for each level of the xvar sum these values across all cells: chisq = ( count - expected)**2/expected; dev = 2*count*(log( count / expected)); Note that if the clogit doesn't work (bad fit or proportional odds not satisfied) then enter link=glogit (in LOGISTIC or GLIMMIX)' the expected and observed counts are the same for a categorical xvar Robin High UNMC From: Tobias Jeppsson <tobias.jeppsson(a)EKOL.SLU.SE> To: SAS-L(a)LISTSERV.UGA.EDU Date: 02/11/2010 05:49 AM Subject: Model fit for ordinal multinomial models in GENMOD? Sent by: "SAS(r) Discussion" <SAS-L(a)LISTSERV.UGA.EDU> Hi I have been fitting models for two response variables, one ordinal and one normal (using AIC to choose the "best" model). For the normal data I can evaluate model fit and assumptions as usual, and by looking at residual plots. However, as I understand it, residuals are not available for ordinal multinomial models in GENMOD. How do I then best assess fit and if I meet the model assumptions? This is the first time that I'm analyzing ordinal multinomial models, so maybe I'm missing something obvious. thanks Tobias Jeppsson
From: Shawn Haskell on 18 Feb 2010 17:29 On Feb 17, 12:18 pm, tobias.jepps...(a)EKOL.SLU.SE (Tobias Jeppsson) wrote: > Thank you both Robin & Peter for your helpful responses. > > When looking at the Score Test for the Proportional Odds Assumption (as Robin suggested) it is indeed significant, indicating a violation of the assumptions. However, when trying Peter's suggestion of comparing the nominal and ordinal formulations I find a relatively good correspondence between predictions, when plotting the predictions for all response classes (R2=0.93). > > To test the "robustness" of the results I also ran the same model assuming dist=poisson (in genmod), even though the assumption of an ordinal response feels more natural. The response categories are (0, 1, 2, 3), with a decreasing number of observation in each category going from 0-3. This analysis come to the same basic conclusion about the effects of my predictors (direction and significance). > > So basically I feel quite confident in the overall results/patterns in my data, but uneasy about violating the basic assumptions in the ordinal model. What do you think is the best way to proceed? > > I'm not certain on how to use the AGGREGATE option to assess fit since I have several predictor variables, both class and continuous. Should I assess fit sequentially by looking at the predictor variables one by one? > > And just to clarify, my response variable is threat classifications of species, with most species considered non-threatened and less species in the higher threat categories. > > Thanks again > > Tobias > > > > -----Original Message----- > From: SAS(r) Discussion [mailto:SA...(a)LISTSERV.UGA.EDU] On Behalf Of Robin R High > Sent: den 12 februari 2010 19:51 > To: SA...(a)LISTSERV.UGA.EDU > Subject: Re: Model fit for ordinal multinomial models in GENMOD? > > Tobias, > > It is somewhat of a multi-step process, namely first determine if the > cumulative logit model assumption is OK with LOGISTIC: > > PROC LOGISTIC DATA=indat; > CLASS xvar / PARAM=glm; > FREQ count; > MODEL y = xvar / link=clogit ; > run; > > Near the top you'll see: > > The LOGISTIC Procedure > > Score Test for the Proportional Odds Assumption > > Chi-Square DF Pr > ChiSq > > 3.9106 3 0.2713 > > which the relatively "large" pvalue indicates the cumulative logit model > works > > To assess goodness of fit for the clogit model and a categorical > explanatory variable, you can run GENMOD with the aggregate= option > > PROC genmod DATA=indat; > CLASS xvar ; > FREQ count; > MODEL y = xvar / dist=multinomial link=clogit aggregate=(xvar) ; > RUN; > > that produces the Criteria For Assessing Goodness Of Fit > > Criterion DF Value Value/DF > > Deviance 3 3.6877 1.2292 > Pearson Chi-Square 3 3.6630 1.2210 > > which are then compared to a chi-square distribution with the indicated > degrees of freedom (DF). > > One can verify these fit statistics by computing the expected counts for > each cell of the table (takes a few programming steps) and connect them > with the observed counts entering them into the chisq and dev formulas: > > expected = total * prob ; where prob is the estimated probability for > each response level for each level of xvar > total is the sum of all observed counts > for each level of the xvar > > sum these values across all cells: > > chisq = ( count - expected)**2/expected; > dev = 2*count*(log( count / expected)); > > Note that if the clogit doesn't work (bad fit or proportional odds not > satisfied) then enter link=glogit (in LOGISTIC or GLIMMIX)' the expected > and observed counts are the same for a categorical xvar > > Robin High > UNMC > > From: > Tobias Jeppsson <tobias.jepps...(a)EKOL.SLU.SE> > To: > SA...(a)LISTSERV.UGA.EDU > Date: > 02/11/2010 05:49 AM > Subject: > Model fit for ordinal multinomial models in GENMOD? > Sent by: > "SAS(r) Discussion" <SA...(a)LISTSERV.UGA.EDU> > > Hi > > I have been fitting models for two response variables, one ordinal and one > normal (using AIC to choose the "best" model). For the normal data I can > evaluate model fit and assumptions as usual, and by looking at residual > plots. However, as I understand it, residuals are not available for > ordinal > multinomial models in GENMOD. How do I then best assess fit and if I meet > the model assumptions? This is the first time that I'm analyzing ordinal > multinomial models, so maybe I'm missing something obvious. > > thanks > Tobias Jeppsson- Hide quoted text - > > - Show quoted text - I had never used ordinal regression until recently, but did something similar recently with moose corpora lutea data (assessing reproductive rates). My ordinal response was identical to yours (0-3). My first choice was Poisson, but the models turned out to be tremendously under- dispersed, and zero-inflated models were not really appropriate. Then I ran the ordinal models - AICc was much improved. Overall, the effects of interest did not really change between types of models, but the ordinal response was a much better fit as evidenced by AICc. If your case is the same, do folks on this listserv agree that this is worth reporting? Shawn
|
Pages: 1 Prev: SAS/IntrNet not cleaning up sas.exe Next: Prevalence vs Incidence |