8 min read

Comparing Marginal effects with margins command

Comparing Marginal effects

Stata’s margins command has been a powerful tool for many economists. It can calculate predicted means as well as predicted marginal effects. Sometimes we’d like to compare those marginal effects. People use margins and marginsplot to generate marginal effects; then draw conclusions on whether there is a difference between marginal effects, based on whether the confidence intervals overlap or not. However, that can actually be wrong. In this post, I’d like to introduce a way to compare effects.

For example:

webuse nhanes2f, clear
logit diabetes i.female##c.age, nolog
margins female,  at(age=(20 30 40 50 60 70)) 
marginsplot
graph export "marginsplot1.svg", as(svg) replace
margins r.female,  at(age=(20 30 40 50 60 70)) 
marginsplot
graph export "marginsplot2.svg", as(svg) replace
Logistic regression                                     Number of obs = 10,335
                                                        LR chi2(3)    = 353.06
                                                        Prob > chi2   = 0.0000
Log likelihood = -1822.5393                             Pseudo R2     = 0.0883

------------------------------------------------------------------------------
    diabetes | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
      female |
     Female  |   1.404111   .4858776     2.89   0.004     .4518088    2.356414
         age |   .0714166   .0063231    11.29   0.000     .0590235    .0838097
             |
female#c.age |
     Female  |  -.0206572   .0078369    -2.64   0.008    -.0360172   -.0052972
             |
       _cons |  -7.041892   .3967299   -17.75   0.000    -7.819468   -6.264315
------------------------------------------------------------------------------


Adjusted predictions                                    Number of obs = 10,335
Model VCE: OIM

Expression: Pr(diabetes), predict()
1._at: age = 20
2._at: age = 30
3._at: age = 40
4._at: age = 50
5._at: age = 60
6._at: age = 70

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
  _at#female |
     1#Male  |   .0036348   .0009895     3.67   0.000     .0016953    .0055743
   1#Female  |   .0097316   .0018436     5.28   0.000     .0061184    .0133449
     2#Male  |    .007396   .0015622     4.73   0.000     .0043341    .0104579
   2#Female  |   .0160637   .0023434     6.85   0.000     .0114706    .0206568
     3#Male  |   .0149906   .0022831     6.57   0.000     .0105159    .0194653
   3#Female  |    .026406   .0027757     9.51   0.000     .0209657    .0318462
     4#Male  |   .0301469   .0029996    10.05   0.000     .0242677    .0360261
   4#Female  |    .043115   .0030923    13.94   0.000     .0370543    .0491758
     5#Male  |   .0596983   .0040248    14.83   0.000     .0518099    .0675867
   5#Female  |    .069641   .0040284    17.29   0.000     .0617454    .0775366
     6#Male  |   .1147889   .0089467    12.83   0.000     .0972536    .1323242
   6#Female  |   .1106004   .0078699    14.05   0.000     .0951757     .126025
------------------------------------------------------------------------------


Variables that uniquely identify margins: age female

file marginsplot1.svg saved as SVG format


Contrasts of adjusted predictions                       Number of obs = 10,335
Model VCE: OIM

Expression: Pr(diabetes), predict()
1._at: age = 20
2._at: age = 30
3._at: age = 40
4._at: age = 50
5._at: age = 60
6._at: age = 70

-------------------------------------------------------
                    |         df        chi2     P>chi2
--------------------+----------------------------------
         female@_at |
(Female vs Male) 1  |          1        8.49     0.0036
(Female vs Male) 2  |          1        9.47     0.0021
(Female vs Male) 3  |          1       10.09     0.0015
(Female vs Male) 4  |          1        9.06     0.0026
(Female vs Male) 5  |          1        3.05     0.0808
(Female vs Male) 6  |          1        0.12     0.7252
             Joint  |          4       40.48     0.0000
-------------------------------------------------------

---------------------------------------------------------------------
                    |            Delta-method
                    |   Contrast   std. err.     [95% conf. interval]
--------------------+------------------------------------------------
         female@_at |
(Female vs Male) 1  |   .0060969   .0020923      .0019959    .0101978
(Female vs Male) 2  |   .0086677   .0028164      .0031476    .0141878
(Female vs Male) 3  |   .0114154    .003594      .0043712    .0184595
(Female vs Male) 4  |   .0129682   .0043081      .0045244    .0214119
(Female vs Male) 5  |   .0099427   .0056945     -.0012183    .0211037
(Female vs Male) 6  |  -.0041885   .0119155     -.0275425    .0191654
---------------------------------------------------------------------


Variables that uniquely identify margins: age

file marginsplot2.svg saved as SVG format

Here we run a logit model of diabetes and have female indicator and continuous age variable and their interaction as predictors.

See my other post about interpreting marginal effects in non-linear model. In general we recommend against going after marginal effects in a non-linear model, instead we should focus on original coefficients. However, in case you still want to do that (many people do), here is an example. Suppose you are interested in comparing the effect of female on probability, at different ages, ranging from 20 to 70. The first margins command above tells you the predicted probability for each gender, the second tells the difference between the two genders.

People tend to draw conclusions based on the confidence intervals overlapping or not. That is, if the confidence intervals of two effects are overlapped, then there is no statistically significant difference between the two effects. This actually can be wrong. The reason is that the confidence intervals are based on individual variable’s distribution, that is, how big the variance of that estimate. The difference between two effects, however, depends on the joint distribution of the two effects. That is, the variances of the two effects, and the covariance between the two effects. We should expect two effects are correlated, since they are derived from estimates from the same model, and often the same data. Therefore, two effects with overlapping confidence intervals do not necessarily mean the difference between the two is not statistically different from zero.

To see this, suppose we are interested in \(x-y\), the difference between two effects \(x\) and \(y\), \[ Var(x-y)=Var(x) + Var(y) - 2 Cov(x,y) \] The variance of the difference is the sum of the two variances, minus the covariance of \(x\) and \(y\). When we draw confidence intervals of marginal effects, \(x\) and \(y\), they are determined by \(Var(x)\) and \(Var(y)\), respectively. But the confidence interval of \(x-y\) should be based on \(Var(x-y)\), which is affected by \(Cov(x,y)\). When \(x\) and \(y\) are highly correlated, the variance of the difference can become small, and therefore the estimated \(x-y\) can be statistically significant, even if the two confidence intervals overlap (meaning the variances of \(x\) and \(y\) can be relatively large).

For example, we see in the second margins output, the effect of female on probability is .006 at point 1 (age 20), .0087 at point 2 (age 30). If you’d like to compare these two effects, you see their confidence intervals overlap. It’s easy to draw a conclusion that these two effects do not differ statistically because their confidence intervals overlap. However, if we do a statistical test of the difference, we see the difference is significant.

webuse nhanes2f, clear
logit diabetes i.female##c.age, nolog
margins r.female,  at(age=(20 30 40 50 60 70)) post
mlincom 2 -1, stat(est se p)
Logistic regression                                     Number of obs = 10,335
                                                        LR chi2(3)    = 353.06
                                                        Prob > chi2   = 0.0000
Log likelihood = -1822.5393                             Pseudo R2     = 0.0883

------------------------------------------------------------------------------
    diabetes | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
      female |
     Female  |   1.404111   .4858776     2.89   0.004     .4518088    2.356414
         age |   .0714166   .0063231    11.29   0.000     .0590235    .0838097
             |
female#c.age |
     Female  |  -.0206572   .0078369    -2.64   0.008    -.0360172   -.0052972
             |
       _cons |  -7.041892   .3967299   -17.75   0.000    -7.819468   -6.264315
------------------------------------------------------------------------------


Contrasts of adjusted predictions                       Number of obs = 10,335
Model VCE: OIM

Expression: Pr(diabetes), predict()
1._at: age = 20
2._at: age = 30
3._at: age = 40
4._at: age = 50
5._at: age = 60
6._at: age = 70

-------------------------------------------------------
                    |         df        chi2     P>chi2
--------------------+----------------------------------
         female@_at |
(Female vs Male) 1  |          1        8.49     0.0036
(Female vs Male) 2  |          1        9.47     0.0021
(Female vs Male) 3  |          1       10.09     0.0015
(Female vs Male) 4  |          1        9.06     0.0026
(Female vs Male) 5  |          1        3.05     0.0808
(Female vs Male) 6  |          1        0.12     0.7252
             Joint  |          4       40.48     0.0000
-------------------------------------------------------

---------------------------------------------------------------------
                    |            Delta-method
                    |   Contrast   std. err.     [95% conf. interval]
--------------------+------------------------------------------------
         female@_at |
(Female vs Male) 1  |   .0060969   .0020923      .0019959    .0101978
(Female vs Male) 2  |   .0086677   .0028164      .0031476    .0141878
(Female vs Male) 3  |   .0114154    .003594      .0043712    .0184595
(Female vs Male) 4  |   .0129682   .0043081      .0045244    .0214119
(Female vs Male) 5  |   .0099427   .0056945     -.0012183    .0211037
(Female vs Male) 6  |  -.0041885   .0119155     -.0275425    .0191654
---------------------------------------------------------------------

             |   lincom        se    pvalue 
-------------+------------------------------
           1 |    0.003     0.001     0.001 

This “mlincom” is a user-written command to test the difference between point 2 and 1 (age 20 and 30).

Fortunately, margins has an option to show this difference, by contrast(atcontrast(r)). This is to say comparing all other points to the reference category.

webuse nhanes2f, clear
logit diabetes i.female##c.age, nolog
margins r.female,  at(age=(20 30 40 50 60 70)) contrast(atcontrast(r))
marginsplot
graph export "marginsplot3.svg", as(svg) replace
Logistic regression                                     Number of obs = 10,335
                                                        LR chi2(3)    = 353.06
                                                        Prob > chi2   = 0.0000
Log likelihood = -1822.5393                             Pseudo R2     = 0.0883

------------------------------------------------------------------------------
    diabetes | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
      female |
     Female  |   1.404111   .4858776     2.89   0.004     .4518088    2.356414
         age |   .0714166   .0063231    11.29   0.000     .0590235    .0838097
             |
female#c.age |
     Female  |  -.0206572   .0078369    -2.64   0.008    -.0360172   -.0052972
             |
       _cons |  -7.041892   .3967299   -17.75   0.000    -7.819468   -6.264315
------------------------------------------------------------------------------


Contrasts of adjusted predictions                       Number of obs = 10,335
Model VCE: OIM

Expression: Pr(diabetes), predict()
1._at: age = 20
2._at: age = 30
3._at: age = 40
4._at: age = 50
5._at: age = 60
6._at: age = 70

--------------------------------------------------------------
                           |         df        chi2     P>chi2
---------------------------+----------------------------------
                _at#female |
(2 vs 1) (Female vs Male)  |          1       10.47     0.0012
(3 vs 1) (Female vs Male)  |          1        9.28     0.0023
(4 vs 1) (Female vs Male)  |          1        5.48     0.0192
(5 vs 1) (Female vs Male)  |          1        0.50     0.4807
(6 vs 1) (Female vs Male)  |          1        0.65     0.4203
                    Joint  |          4       20.04     0.0005
--------------------------------------------------------------

----------------------------------------------------------------------------
                           |            Delta-method
                           |   Contrast   std. err.     [95% conf. interval]
---------------------------+------------------------------------------------
                _at#female |
(2 vs 1) (Female vs Male)  |   .0025709   .0007946      .0010134    .0041283
(3 vs 1) (Female vs Male)  |   .0053185    .001746      .0018964    .0087407
(4 vs 1) (Female vs Male)  |   .0068713   .0029351      .0011187    .0126239
(5 vs 1) (Female vs Male)  |   .0038458   .0054532     -.0068422    .0145339
(6 vs 1) (Female vs Male)  |  -.0102854   .0127614     -.0352972    .0147264
----------------------------------------------------------------------------


Variables that uniquely identify margins: age

file marginsplot3.svg saved as SVG format

From this we see the difference of effects between point 2 and 1 is .0026, and significant, the same conclusion as mlincom shows.