Example 9.1 (Simpson’s paradox). There are two doctors, Dr. Lee and Dr. Wong, performing two types of surgeries — heart surgery (hard) and band-aid removal (easy). Dr. Lee has higher overall surgery success rate. Is Dr. Lee necessarily a better doctor than Dr. Wong?
No. Consider the following example:
Dr. Lee
Dr. Wong
Heart
Band-Aid
Total
Heart
Band-Aid
Total
Success
2
81
83
70
10
80
Failure
8
9
17
20
0
20
Success rate
20%
90%
83%
78%
100%
80%
The truth is Dr. Lee has overall higher success rate because he only does easy surgeries (band-aid removal). Dr. Wong does mostly hard surgeries and thus has a lower overall success rate. Yet, he is better at each single type of surgery. To formalize the argument, let \(S\): successful surgery; \(D\): treated by Dr. Lee, \(D^{c}\): treated by Dr. Wong; \(E\): heart surgery, \(E^{c}\): band-aid removal. Dr. Wong is better at each type of surgery, \[\begin{aligned}P(S|D,E) & <P(S|D^{c},E)\\
P(S|D,E^{c}) & <P(S|D^{c},E^{c});
\end{aligned}\] But, Dr. Lee has a higher overall successful rate, \[P(S|D)>P(S|D^{c}).\] This is because there is a “confounder” \(E\): \[P(S|D)=\underbrace{P(S|D,E)}_{<P(S|D^{c},E)}\underbrace{P(E|D)}_{\textrm{weight}}+\underbrace{P(S|D,E^{c})}_{<P(S|D^{c},E^{c})}\underbrace{P(E^{c}|D)}_{\textrm{weight}}.\]
A confounder is a variable that influences with both explanatory variable and the outcome variable, which therefore “confounds” the correlation between the two. In our example, the type of surgery (\(E\)) is associated with both the doctor and the outcome. Without the confounder being controlled, it is impossible to draw valid conclusions from the statistics.
In general terms, Simpson’s paradox refers to the paradox in which a trend that appears across different groups of aggregate data is the reverse of the trend that appears when the aggregate data is broken up into its components. It is one of the most common sources of statistical misuse. Here is another example.
Example 9.2 (UC Berkeley gender bias). One of the best-known examples of Simpson’s paradox comes from a study of gender bias among graduate school admissions to University of California, Berkeley. The admission figures for the fall of 1973 showed that men applying were more likely than women to be admitted, and the difference was so large that it was unlikely to be due to chance.
Male
Female
Applicants
Admitted
Applicants
Admitted
Total
8,442
44%
4,321
35%
However, when taking into account the information about departments being applied to, the conclusion turns to the opposite: in most departments, the admission rate for women is higher than men. The lower overall admission rate is caused by the fact that women tended to apply to more competitive departments with lower rates of admission, whereas men tended to apply to less competitive departments with higher rates of admission.
# R has a built-in dataset `UCBAdmissions`# we convert it to data frame for analysisdata <-as.data.frame(UCBAdmissions)# browse the first a few rowshead(data)
Admit Gender Dept Freq
1 Admitted Male A 512
2 Rejected Male A 313
3 Admitted Female A 89
4 Rejected Female A 19
5 Admitted Male B 353
6 Rejected Male B 207
# subset of the data with only admissionsdata <-subset(data, Admit =='Admitted')# number of admissions by Genderaggregate(Freq ~ Gender, data = data, FUN = sum)
Gender Freq
1 Male 1198
2 Female 557
# number of admissions by Gender and Departmentaggregate(Freq ~ Gender + Dept, data = data, FUN = sum)
Gender Dept Freq
1 Male A 512
2 Female A 89
3 Male B 353
4 Female B 17
5 Male C 120
6 Female C 202
7 Male D 138
8 Female D 131
9 Male E 53
10 Female E 94
11 Male F 22
12 Female F 24
The importance of conditional thinking
Whenever we talk about probability or statistics, always remind ourselves what we are the conditioning on. Any statistical reasoning without specifying the conditions can be misleading. We are prone to such fallacies everyday everywhere.
“10 millions new jobs were added during the term of President X.” But it doesn’t tell you this was achieved conditioned on that the economy had just had the worst recession.
“Private schools’ graduates earned 50% more than those graduated from public schools.” But it doesn’t tell you the background of those students who enrolled in private schools.
Be vigilant to these claims when you see them next time.