Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

L2-06 · See a reversed comparison

One option can win inside every group and still lose when the groups are added together. This happens when the options get very different mixes of easy and hard cases. Which comparison to trust depends on the causal story behind the numbers.

Level 2 icon Level 2 · Grades 7–9 · about 45 minutes
Big idea. One option can win inside every group and still lose when the groups are added together. This happens when the options get very different mixes of easy and hard cases. Which comparison to trust depends on the causal story behind the numbers.
Before this lesson. L2-04, L1-06
Grown-ups. The guide for adults is at the end of this page.

What you will learn

Story: Which tutoring program works better?

Leo’s imagined school tries two tutoring programs, A and B. Each program helps 100 students. All the numbers here are made up. Some students take easy courses and some take hard courses. Leo counts who passed.

In easy courses, program A passed 18 of 20 students, which is 90 percent. Program B passed 64 of 80, which is 80 percent. In hard courses, A passed 32 of 80, which is 40 percent. B passed 6 of 20, which is 30 percent. So A is ahead in easy courses and in hard courses.

Then Maya adds the groups together. Program A passed 18 + 32 = 50 of 100, which is 50 percent. Program B passed 64 + 6 = 70 of 100, which is 70 percent. Now B looks much better! The comparison flipped. This is called a Simpson’s reversal.

The reason is the mix of students. Most of A’s students, 80 of 100, were in hard courses, where everyone passes less often. Most of B’s students were in easy courses. So the combined rate mostly measures which courses the students took. Leo draws a causal map. Course difficulty points to the program and to passing. Difficulty came first and affects both, so if this map is right, the fair test compares A and B within each group. Then A wins. With a different causal story, the combined numbers could be the right ones.

A real case like this was published in 1975 about graduate school admissions at the University of California, Berkeley. Across the whole university, men were admitted at a higher rate than women. Inside most departments, the gap vanished or slightly favored women. Women had applied more often to the departments that admitted a small share of everyone who applied.

Program A wins in each group but loses when the groups are combined.

What the animation shows. Bars show made-up pass rates for two tutoring programs. In easy courses, A passed 18 of 20, or 90 percent, and B passed 64 of 80, or 80 percent. In hard courses, A passed 32 of 80, or 40 percent, and B passed 6 of 20, or 30 percent. A is ahead in both. Next, each bar becomes as wide as its number of students. Each program’s two bars slide together and merge into one bar with the same area. Combined, A passed 50 of 100, or 50 percent, and B passed 70 of 100, or 70 percent. Last, a causal map shows course difficulty with arrows to the program and to passing. A third arrow, from the program to passing, has a question mark. The closing lines say that difficulty affects both, so here we compare A and B within each group.

Hands-on: Find the flip, then build your own

About 20 minutes.

You need:

Steps:

  1. Here is a made-up table about two imagined pizza shops, P and Q. On sunny days, P is on time for 45 of 50 orders and Q for 136 of 160. On rainy days, P is on time for 75 of 150 orders and Q for 16 of 40.

  2. Find each shop’s on-time rate on sunny days and on rainy days. Which shop is ahead in each kind of weather?

  3. Find each shop’s combined rate. Add its on-time orders and divide by all its orders. Which shop is ahead now?

  4. Explain the flip in one sentence. Look at how many rainy-day orders each shop had.

  5. Now build your own reversal with counters. One color means success, the other means failure. Make two groups, easy and hard, for two options, X and Y.

  6. Arrange the counters so X beats Y in both groups. Then give X mostly hard cases and Y mostly easy cases.

  7. Count the combined rates. If X is not behind yet, move more of X’s cases into the hard group and try again.

  8. Draw a causal map for your story. Which comparison would you trust, and why?

Practice together

Use Leo’s made-up numbers. Why can you not just average A’s two rates, 90 percent and 40 percent, to get A’s combined rate?

Check your answer

The average of 90 and 40 is 65, but that treats both groups as the same size. A had only 20 students in easy courses and 80 in hard courses. Counting students gives 50 of 100, which is 50 percent. The big hard group pulls A’s combined rate down.

Try it yourself

1. Two imagined players take shots (made-up numbers). On close shots, A scores 8 of 10 and B scores 35 of 50. On far shots, A scores 20 of 50 and B scores 3 of 10. Who is better at each kind of shot? Who is better combined?

Check your answer

Close: A has 80 percent and B has 70 percent. Far: A has 40 percent and B has 30 percent. So A is better at both. Combined, A scores 28 of 60, about 47 percent, and B scores 38 of 60, about 63 percent. A took far more of the hard, far shots.

2. In the pizza table, would you choose a shop by the combined rate or by the weather rates? Use a causal map to explain.

Check your answer

Rain makes every shop late more often, and P got far more of its orders on rainy days. So weather sits behind both the mix of orders and being on time. On any given kind of day, the weather rates are the fair comparison, and P wins both.

Talk about it

Check yourself

1. In Leo’s made-up data, what is program B’s combined pass rate?

Show the answer

C. 70 of 100

B passed 64 students in easy courses and 6 in hard courses. That is 70 of 100 students in all.

2. Why did program A look worse when the groups were combined?

Show the answer

B. Most of A’s students took hard courses.

A was ahead in both groups. But 80 of its 100 students were in hard courses, where pass rates are lower for everyone.

3. Remember Lesson 4. In a made-up table, 12 of 30 students in a club also play chess. What is P(chess | club)? (Review of L2-04)

Show the answer

B. 12 out of 30

The condition is “in the club”, so the 30 club members are the whole. Of them, 12 play chess.

Words to know

Simpson’s reversal
When a comparison points one way inside every group but the other way once the groups are combined.
combined rate
All the successes from every group divided by all the cases from every group.
causal map
A drawing with arrows from causes to the things they affect. It shows what we assume, not a proven fact.
For grown-ups: about this lesson

Why it matters. The lesson shows Simpson’s reversal with made-up numbers chosen so that every rate is a whole percent and both programs have 100 students: easy 18/20 vs 64/80, hard 32/80 vs 6/20, combined 50/100 vs 70/100. Bickel, Hammel and O’Connell analyzed Berkeley graduate admissions and found that the aggregate gap favoring men mostly disappeared within departments, because women applied more to departments with low admission rates Bickel et al., 1975; the lesson describes this in words only. Pearl’s overview explains that whether to adjust for a variable depends on the assumed causal model, which is why the choice between split and combined data cannot be made from the numbers alone Pearl, 2009. Cunningham’s text covers causal comparisons, confounders and research design at a level suitable for further reading by teachers Cunningham, 2021. Here difficulty is a common cause measured before the program, so the split comparison is the fair one under that assumed map.

Watch for these mix-ups.

  • The combined numbers are always the true answer. (They mix in differences in who got which option; with a common cause, compare within groups.)

  • The split numbers are always the true answer. (If the program itself changes the grouping variable, splitting can hide part of its effect; the causal map decides.)

  • A reversal means someone made an arithmetic mistake. (The arithmetic is right; unequal group sizes produce the flip.)

  • The Berkeley data proved there was no bias anywhere. (It showed the overall gap was mostly explained by where people applied; it did not settle why women applied where they did.)

To make it easier. Work only with Leo’s table. Give the four group rates and ask learners to add the counts for the two combined rates, then say which program looks better each way.

To make it harder. Find the smallest change to Leo’s table that stops the reversal. Then read about the 1975 Berkeley study and explain, without numbers, why department choice acts like course difficulty.

Limits. All tutoring, pizza and shooting numbers are invented. The causal map is an assumption; if other causes of passing differ between programs, even the split comparison may not show the program’s effect.

On the computer. Grown-ups can see where this idea leads later in Lab: Confounding and intervention, written for older learners.

Sources for this guide. Bickel et al., 1975Pearl, 2009Cunningham, 2021

Next: L2-07 · Read a causal graph

References
  1. Bickel, P. J., Hammel, E. A., & O’Connell, J. W. (1975). Sex Bias in Graduate Admissions: Data from Berkeley. Science, 187(4175), 398–404. 10.1126/science.187.4175.398
  2. Pearl, J. (2009). Causal inference in statistics: An overview. Statistics Surveys, 3(none), 96–146. 10.1214/09-ss057
  3. Cunningham, S. (2021). Causal Inference: The Mixtape. Yale University Press. 10.12987/9780300255881