Comparing Marginal effects
Stata’s margins command has been a powerful tool for many economists. It can calculate predicted means as well as predicted marginal effects. Sometimes we’d like to compare those marginal effects. People use margins and marginsplot to generate marginal effects; then draw conclusions on whether there is a difference between marginal effects, based on whether the confidence intervals overlap or not. However, that can actually be wrong. In this post, I’d like to introduce a way to compare effects.
For example:
webuse nhanes2f, clear
logit diabetes i.female##c.age, nolog
margins female, at(age=(20 30 40 50 60 70))
marginsplot
graph export "marginsplot1.svg", as(svg) replace
margins r.female, at(age=(20 30 40 50 60 70))
marginsplot
graph export "marginsplot2.svg", as(svg) replaceLogistic regression Number of obs = 10,335
LR chi2(3) = 353.06
Prob > chi2 = 0.0000
Log likelihood = -1822.5393 Pseudo R2 = 0.0883
------------------------------------------------------------------------------
diabetes | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
female |
Female | 1.404111 .4858776 2.89 0.004 .4518088 2.356414
age | .0714166 .0063231 11.29 0.000 .0590235 .0838097
|
female#c.age |
Female | -.0206572 .0078369 -2.64 0.008 -.0360172 -.0052972
|
_cons | -7.041892 .3967299 -17.75 0.000 -7.819468 -6.264315
------------------------------------------------------------------------------
Adjusted predictions Number of obs = 10,335
Model VCE: OIM
Expression: Pr(diabetes), predict()
1._at: age = 20
2._at: age = 30
3._at: age = 40
4._at: age = 50
5._at: age = 60
6._at: age = 70
------------------------------------------------------------------------------
| Delta-method
| Margin std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
_at#female |
1#Male | .0036348 .0009895 3.67 0.000 .0016953 .0055743
1#Female | .0097316 .0018436 5.28 0.000 .0061184 .0133449
2#Male | .007396 .0015622 4.73 0.000 .0043341 .0104579
2#Female | .0160637 .0023434 6.85 0.000 .0114706 .0206568
3#Male | .0149906 .0022831 6.57 0.000 .0105159 .0194653
3#Female | .026406 .0027757 9.51 0.000 .0209657 .0318462
4#Male | .0301469 .0029996 10.05 0.000 .0242677 .0360261
4#Female | .043115 .0030923 13.94 0.000 .0370543 .0491758
5#Male | .0596983 .0040248 14.83 0.000 .0518099 .0675867
5#Female | .069641 .0040284 17.29 0.000 .0617454 .0775366
6#Male | .1147889 .0089467 12.83 0.000 .0972536 .1323242
6#Female | .1106004 .0078699 14.05 0.000 .0951757 .126025
------------------------------------------------------------------------------
Variables that uniquely identify margins: age female
file marginsplot1.svg saved as SVG format
Contrasts of adjusted predictions Number of obs = 10,335
Model VCE: OIM
Expression: Pr(diabetes), predict()
1._at: age = 20
2._at: age = 30
3._at: age = 40
4._at: age = 50
5._at: age = 60
6._at: age = 70
-------------------------------------------------------
| df chi2 P>chi2
--------------------+----------------------------------
female@_at |
(Female vs Male) 1 | 1 8.49 0.0036
(Female vs Male) 2 | 1 9.47 0.0021
(Female vs Male) 3 | 1 10.09 0.0015
(Female vs Male) 4 | 1 9.06 0.0026
(Female vs Male) 5 | 1 3.05 0.0808
(Female vs Male) 6 | 1 0.12 0.7252
Joint | 4 40.48 0.0000
-------------------------------------------------------
---------------------------------------------------------------------
| Delta-method
| Contrast std. err. [95% conf. interval]
--------------------+------------------------------------------------
female@_at |
(Female vs Male) 1 | .0060969 .0020923 .0019959 .0101978
(Female vs Male) 2 | .0086677 .0028164 .0031476 .0141878
(Female vs Male) 3 | .0114154 .003594 .0043712 .0184595
(Female vs Male) 4 | .0129682 .0043081 .0045244 .0214119
(Female vs Male) 5 | .0099427 .0056945 -.0012183 .0211037
(Female vs Male) 6 | -.0041885 .0119155 -.0275425 .0191654
---------------------------------------------------------------------
Variables that uniquely identify margins: age
file marginsplot2.svg saved as SVG format
Here we run a logit model of diabetes and have female indicator and continuous age variable and their interaction as predictors.
See my other post about interpreting marginal effects in non-linear model. In general we recommend against going after marginal effects in a non-linear model, instead we should focus on original coefficients. However, in case you still want to do that (many people do), here is an example. Suppose you are interested in comparing the effect of female on probability, at different ages, ranging from 20 to 70. The first margins command above tells you the predicted probability for each gender, the second tells the difference between the two genders.
People tend to draw conclusions based on the confidence intervals overlapping or not. That is, if the confidence intervals of two effects are overlapped, then there is no statistically significant difference between the two effects. This actually can be wrong. The reason is that the confidence intervals are based on individual variable’s distribution, that is, how big the variance of that estimate. The difference between two effects, however, depends on the joint distribution of the two effects. That is, the variances of the two effects, and the covariance between the two effects. We should expect two effects are correlated, since they are derived from estimates from the same model, and often the same data. Therefore, two effects with overlapping confidence intervals do not necessarily mean the difference between the two is not statistically different from zero.
To see this, suppose we are interested in \(x-y\), the difference between two effects \(x\) and \(y\), \[ Var(x-y)=Var(x) + Var(y) - 2 Cov(x,y) \] The variance of the difference is the sum of the two variances, minus the covariance of \(x\) and \(y\). When we draw confidence intervals of marginal effects, \(x\) and \(y\), they are determined by \(Var(x)\) and \(Var(y)\), respectively. But the confidence interval of \(x-y\) should be based on \(Var(x-y)\), which is affected by \(Cov(x,y)\). When \(x\) and \(y\) are highly correlated, the variance of the difference can become small, and therefore the estimated \(x-y\) can be statistically significant, even if the two confidence intervals overlap (meaning the variances of \(x\) and \(y\) can be relatively large).
For example, we see in the second margins output, the effect of female on probability is .006 at point 1 (age 20), .0087 at point 2 (age 30). If you’d like to compare these two effects, you see their confidence intervals overlap. It’s easy to draw a conclusion that these two effects do not differ statistically because their confidence intervals overlap. However, if we do a statistical test of the difference, we see the difference is significant.
webuse nhanes2f, clear
logit diabetes i.female##c.age, nolog
margins r.female, at(age=(20 30 40 50 60 70)) post
mlincom 2 -1, stat(est se p)Logistic regression Number of obs = 10,335
LR chi2(3) = 353.06
Prob > chi2 = 0.0000
Log likelihood = -1822.5393 Pseudo R2 = 0.0883
------------------------------------------------------------------------------
diabetes | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
female |
Female | 1.404111 .4858776 2.89 0.004 .4518088 2.356414
age | .0714166 .0063231 11.29 0.000 .0590235 .0838097
|
female#c.age |
Female | -.0206572 .0078369 -2.64 0.008 -.0360172 -.0052972
|
_cons | -7.041892 .3967299 -17.75 0.000 -7.819468 -6.264315
------------------------------------------------------------------------------
Contrasts of adjusted predictions Number of obs = 10,335
Model VCE: OIM
Expression: Pr(diabetes), predict()
1._at: age = 20
2._at: age = 30
3._at: age = 40
4._at: age = 50
5._at: age = 60
6._at: age = 70
-------------------------------------------------------
| df chi2 P>chi2
--------------------+----------------------------------
female@_at |
(Female vs Male) 1 | 1 8.49 0.0036
(Female vs Male) 2 | 1 9.47 0.0021
(Female vs Male) 3 | 1 10.09 0.0015
(Female vs Male) 4 | 1 9.06 0.0026
(Female vs Male) 5 | 1 3.05 0.0808
(Female vs Male) 6 | 1 0.12 0.7252
Joint | 4 40.48 0.0000
-------------------------------------------------------
---------------------------------------------------------------------
| Delta-method
| Contrast std. err. [95% conf. interval]
--------------------+------------------------------------------------
female@_at |
(Female vs Male) 1 | .0060969 .0020923 .0019959 .0101978
(Female vs Male) 2 | .0086677 .0028164 .0031476 .0141878
(Female vs Male) 3 | .0114154 .003594 .0043712 .0184595
(Female vs Male) 4 | .0129682 .0043081 .0045244 .0214119
(Female vs Male) 5 | .0099427 .0056945 -.0012183 .0211037
(Female vs Male) 6 | -.0041885 .0119155 -.0275425 .0191654
---------------------------------------------------------------------
| lincom se pvalue
-------------+------------------------------
1 | 0.003 0.001 0.001 This “mlincom” is a user-written command to test the difference between point 2 and 1 (age 20 and 30).
Fortunately, margins has an option to show this difference, by contrast(atcontrast(r)). This is to say comparing all other points to the reference category.
webuse nhanes2f, clear
logit diabetes i.female##c.age, nolog
margins r.female, at(age=(20 30 40 50 60 70)) contrast(atcontrast(r))
marginsplot
graph export "marginsplot3.svg", as(svg) replaceLogistic regression Number of obs = 10,335
LR chi2(3) = 353.06
Prob > chi2 = 0.0000
Log likelihood = -1822.5393 Pseudo R2 = 0.0883
------------------------------------------------------------------------------
diabetes | Coefficient Std. err. z P>|z| [95% conf. interval]
-------------+----------------------------------------------------------------
female |
Female | 1.404111 .4858776 2.89 0.004 .4518088 2.356414
age | .0714166 .0063231 11.29 0.000 .0590235 .0838097
|
female#c.age |
Female | -.0206572 .0078369 -2.64 0.008 -.0360172 -.0052972
|
_cons | -7.041892 .3967299 -17.75 0.000 -7.819468 -6.264315
------------------------------------------------------------------------------
Contrasts of adjusted predictions Number of obs = 10,335
Model VCE: OIM
Expression: Pr(diabetes), predict()
1._at: age = 20
2._at: age = 30
3._at: age = 40
4._at: age = 50
5._at: age = 60
6._at: age = 70
--------------------------------------------------------------
| df chi2 P>chi2
---------------------------+----------------------------------
_at#female |
(2 vs 1) (Female vs Male) | 1 10.47 0.0012
(3 vs 1) (Female vs Male) | 1 9.28 0.0023
(4 vs 1) (Female vs Male) | 1 5.48 0.0192
(5 vs 1) (Female vs Male) | 1 0.50 0.4807
(6 vs 1) (Female vs Male) | 1 0.65 0.4203
Joint | 4 20.04 0.0005
--------------------------------------------------------------
----------------------------------------------------------------------------
| Delta-method
| Contrast std. err. [95% conf. interval]
---------------------------+------------------------------------------------
_at#female |
(2 vs 1) (Female vs Male) | .0025709 .0007946 .0010134 .0041283
(3 vs 1) (Female vs Male) | .0053185 .001746 .0018964 .0087407
(4 vs 1) (Female vs Male) | .0068713 .0029351 .0011187 .0126239
(5 vs 1) (Female vs Male) | .0038458 .0054532 -.0068422 .0145339
(6 vs 1) (Female vs Male) | -.0102854 .0127614 -.0352972 .0147264
----------------------------------------------------------------------------
Variables that uniquely identify margins: age
file marginsplot3.svg saved as SVG format
From this we see the difference of effects between point 2 and 1 is .0026, and significant, the same conclusion as mlincom shows.