8 min read

Comparing Marginal effects with margins command

Comparing Marginal effects

Stata’s margins command has been a powerful tool for many economists. It can calculate predicted means as well as predicted marginal effects. Sometimes we’d like to compare those marginal effects. People use margins and marginsplot to generate marginal effects; then draw conclusions on whether there is a difference between marginal effects, based on whether the confidence intervals overlap or not. However, that can actually be wrong. In this post, I’d like to introduce a way to compare effects.

For example:

webuse nhanes2f, clear
logit diabetes i.female##c.age, nolog
margins female,  at(age=(20 30 40 50 60 70)) 
marginsplot
graph export "marginsplot1.svg", as(svg) replace
margins r.female,  at(age=(20 30 40 50 60 70)) 
marginsplot
graph export "marginsplot2.svg", as(svg) replace
Logistic regression                             Number of obs     =     10,335
                                                LR chi2(3)        =     353.06
                                                Prob > chi2       =     0.0000
Log likelihood = -1822.5393                     Pseudo R2         =     0.0883

------------------------------------------------------------------------------
    diabetes |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    1.female |   1.404111   .4858776     2.89   0.004     .4518088    2.356414
         age |   .0714166   .0063231    11.29   0.000     .0590235    .0838097
             |
female#c.age |
          1  |  -.0206572   .0078369    -2.64   0.008    -.0360172   -.0052972
             |
       _cons |  -7.041892   .3967299   -17.75   0.000    -7.819468   -6.264315
------------------------------------------------------------------------------


Adjusted predictions                            Number of obs     =     10,335
Model VCE    : OIM

Expression   : Pr(diabetes), predict()

1._at        : age             =          20

2._at        : age             =          30

3._at        : age             =          40

4._at        : age             =          50

5._at        : age             =          60

6._at        : age             =          70

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
  _at#female |
        1 0  |   .0036348   .0009895     3.67   0.000     .0016953    .0055743
        1 1  |   .0097316   .0018436     5.28   0.000     .0061184    .0133449
        2 0  |    .007396   .0015622     4.73   0.000     .0043341    .0104579
        2 1  |   .0160637   .0023434     6.85   0.000     .0114706    .0206568
        3 0  |   .0149906   .0022831     6.57   0.000     .0105159    .0194653
        3 1  |    .026406   .0027757     9.51   0.000     .0209657    .0318462
        4 0  |   .0301469   .0029996    10.05   0.000     .0242677    .0360261
        4 1  |    .043115   .0030923    13.94   0.000     .0370543    .0491758
        5 0  |   .0596983   .0040248    14.83   0.000     .0518099    .0675867
        5 1  |    .069641   .0040284    17.29   0.000     .0617454    .0775366
        6 0  |   .1147889   .0089467    12.83   0.000     .0972536    .1323242
        6 1  |   .1106004   .0078699    14.05   0.000     .0951757     .126025
------------------------------------------------------------------------------


  Variables that uniquely identify margins: age female

(note: file marginsplot1.svg not found)
(file marginsplot1.svg written in SVG format)


Contrasts of adjusted predictions
Model VCE    : OIM

Expression   : Pr(diabetes), predict()

1._at        : age             =          20

2._at        : age             =          30

3._at        : age             =          40

4._at        : age             =          50

5._at        : age             =          60

6._at        : age             =          70

------------------------------------------------
             |         df        chi2     P>chi2
-------------+----------------------------------
  female@_at |
 (1 vs 0) 1  |          1        8.49     0.0036
 (1 vs 0) 2  |          1        9.47     0.0021
 (1 vs 0) 3  |          1       10.09     0.0015
 (1 vs 0) 4  |          1        9.06     0.0026
 (1 vs 0) 5  |          1        3.05     0.0808
 (1 vs 0) 6  |          1        0.12     0.7252
      Joint  |          4       40.48     0.0000
------------------------------------------------

--------------------------------------------------------------
             |            Delta-method
             |   Contrast   Std. Err.     [95% Conf. Interval]
-------------+------------------------------------------------
  female@_at |
 (1 vs 0) 1  |   .0060969   .0020923      .0019959    .0101978
 (1 vs 0) 2  |   .0086677   .0028164      .0031476    .0141878
 (1 vs 0) 3  |   .0114154    .003594      .0043712    .0184595
 (1 vs 0) 4  |   .0129682   .0043081      .0045244    .0214119
 (1 vs 0) 5  |   .0099427   .0056945     -.0012183    .0211037
 (1 vs 0) 6  |  -.0041885   .0119155     -.0275425    .0191654
--------------------------------------------------------------


  Variables that uniquely identify margins: age

(note: file marginsplot2.svg not found)
(file marginsplot2.svg written in SVG format)

Here we run a logit model of diabetes and have female indicator and continous age variable and their interaction as predictors.

See my other post about interpreting marginal effects in non-linear model. In general we recommend against going after marginal effects in a non-linear model, instead we should focus on original coeffcients. However, in case you still want to do that (many people do), here is an example. Suppose you are interested in comparing the effect of female on probability, at different ages, ranging from 20 to 70. The first margins command above tells you the predicted probability for each gender, the second tells the difference between the two genders.

People tend to draw conclusions based on the confidence intervals overlapping or not. That is, if the confidence intervals of two effects are overlapped, then there is no statistically significant difference between the two effects. This actually can be wrong. The reason is that the confidence intervals are based on indidual variable’s distribution, that is, how big the variance of that estimate. The difference between two effects, however, depends on the joint distribution of the two effects. That is, the variances of the two effects, and the covariance between the two effects. We should expect two effects are correlated, since they are derived from estimates from the same model, and often the same data. Therefore, two effects with overlapping confidence intervals do not necessarily mean the difference between the two is not statistically different from zero. But the reverse is true, that is, if the difference is significant, the two effects should have non-overlapping confidence intervals.

To see this, suppose we are interested in \(x-y\), the difference between two effects \(x\) and \(y\), \[ Var(x-y)=Var(x) + Var(y) - 2 Cov(x,y) \] The variance of the difference is the sum of the two variances, minus the covariance of \(x\) and \(y\). When we draw confidence intervals of marginal effects, \(x\) and \(y\), they are determined by \(Var(x)\) and \(Var(y)\), respectively. But the confidence interval of \(x-y\) should be based on \(Var(x-y)\), which is affected by \(Cov(x,y)\). When \(x\) and \(y\) are highly correlated, the variance of the differnce can become small, and therefore the estimated \(x-y\) can be statisically significant, even if the two confidence intervals overlap (meaning the variances of \(x\) and \(y\) can be relatively large).

For example, we see in the second margins output, the effect of female on probability is .006 at piont 1 (age 20), .0087 at point 2 (age 30). If you’d like to compare these two effects, you see their confidence intervals overlap. It’s easy to draw a conclusion that these two effects do not differ statistically because their confidence intervals overlap. However, if we do a statistical test of the difference, we see the difference is significant.

webuse nhanes2f, clear
logit diabetes i.female##c.age, nolog
margins r.female,  at(age=(20 30 40 50 60 70)) post
mlincom 2 -1, stat(est se p)
Logistic regression                             Number of obs     =     10,335
                                                LR chi2(3)        =     353.06
                                                Prob > chi2       =     0.0000
Log likelihood = -1822.5393                     Pseudo R2         =     0.0883

------------------------------------------------------------------------------
    diabetes |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    1.female |   1.404111   .4858776     2.89   0.004     .4518088    2.356414
         age |   .0714166   .0063231    11.29   0.000     .0590235    .0838097
             |
female#c.age |
          1  |  -.0206572   .0078369    -2.64   0.008    -.0360172   -.0052972
             |
       _cons |  -7.041892   .3967299   -17.75   0.000    -7.819468   -6.264315
------------------------------------------------------------------------------


Contrasts of adjusted predictions
Model VCE    : OIM

Expression   : Pr(diabetes), predict()

1._at        : age             =          20

2._at        : age             =          30

3._at        : age             =          40

4._at        : age             =          50

5._at        : age             =          60

6._at        : age             =          70

------------------------------------------------
             |         df        chi2     P>chi2
-------------+----------------------------------
  female@_at |
 (1 vs 0) 1  |          1        8.49     0.0036
 (1 vs 0) 2  |          1        9.47     0.0021
 (1 vs 0) 3  |          1       10.09     0.0015
 (1 vs 0) 4  |          1        9.06     0.0026
 (1 vs 0) 5  |          1        3.05     0.0808
 (1 vs 0) 6  |          1        0.12     0.7252
      Joint  |          4       40.48     0.0000
------------------------------------------------

--------------------------------------------------------------
             |            Delta-method
             |   Contrast   Std. Err.     [95% Conf. Interval]
-------------+------------------------------------------------
  female@_at |
 (1 vs 0) 1  |   .0060969   .0020923      .0019959    .0101978
 (1 vs 0) 2  |   .0086677   .0028164      .0031476    .0141878
 (1 vs 0) 3  |   .0114154    .003594      .0043712    .0184595
 (1 vs 0) 4  |   .0129682   .0043081      .0045244    .0214119
 (1 vs 0) 5  |   .0099427   .0056945     -.0012183    .0211037
 (1 vs 0) 6  |  -.0041885   .0119155     -.0275425    .0191654
--------------------------------------------------------------

             |   lincom        se    pvalue 
-------------+------------------------------
           1 |    0.003     0.001     0.001 

This “mlincom” is a user-written command to test the difference between point 2 and 1 (age 20 and 30).

Fortunately, margins has an option to show this difference, by contrast(atcontrast(r)). This is to say comparing all other points to the reference category.

webuse nhanes2f, clear
logit diabetes i.female##c.age, nolog
margins r.female,  at(age=(20 30 40 50 60 70)) contrast(atcontrast(r))
marginsplot
graph export "marginsplot3.svg", as(svg) replace
Logistic regression                             Number of obs     =     10,335
                                                LR chi2(3)        =     353.06
                                                Prob > chi2       =     0.0000
Log likelihood = -1822.5393                     Pseudo R2         =     0.0883

------------------------------------------------------------------------------
    diabetes |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    1.female |   1.404111   .4858776     2.89   0.004     .4518088    2.356414
         age |   .0714166   .0063231    11.29   0.000     .0590235    .0838097
             |
female#c.age |
          1  |  -.0206572   .0078369    -2.64   0.008    -.0360172   -.0052972
             |
       _cons |  -7.041892   .3967299   -17.75   0.000    -7.819468   -6.264315
------------------------------------------------------------------------------


Contrasts of adjusted predictions
Model VCE    : OIM

Expression   : Pr(diabetes), predict()

1._at        : age             =          20

2._at        : age             =          30

3._at        : age             =          40

4._at        : age             =          50

5._at        : age             =          60

6._at        : age             =          70

------------------------------------------------------
                   |         df        chi2     P>chi2
-------------------+----------------------------------
        _at#female |
(2 vs 1) (1 vs 0)  |          1       10.47     0.0012
(3 vs 1) (1 vs 0)  |          1        9.28     0.0023
(4 vs 1) (1 vs 0)  |          1        5.48     0.0192
(5 vs 1) (1 vs 0)  |          1        0.50     0.4807
(6 vs 1) (1 vs 0)  |          1        0.65     0.4203
            Joint  |          4       20.04     0.0005
------------------------------------------------------

--------------------------------------------------------------------
                   |            Delta-method
                   |   Contrast   Std. Err.     [95% Conf. Interval]
-------------------+------------------------------------------------
        _at#female |
(2 vs 1) (1 vs 0)  |   .0025709   .0007946      .0010134    .0041283
(3 vs 1) (1 vs 0)  |   .0053185    .001746      .0018964    .0087407
(4 vs 1) (1 vs 0)  |   .0068713   .0029351      .0011187    .0126239
(5 vs 1) (1 vs 0)  |   .0038458   .0054532     -.0068422    .0145339
(6 vs 1) (1 vs 0)  |  -.0102854   .0127614     -.0352972    .0147264
--------------------------------------------------------------------


  Variables that uniquely identify margins: age

(note: file marginsplot3.svg not found)
(file marginsplot3.svg written in SVG format)

From this we see the difference of effects between point 2 and 1 is .0026, and significant, the same conclusion as mlincom shows.