L2-08 · Observe or intervene
Watching who chose something and assigning it to people answer different questions. When a common cause pushes both the choice and the result, the watched gap can be much bigger than the real effect. A coin that decides who gets what removes that push.
Level 2 · Grades 7–9 · about 45 minutes
Big idea. Watching who chose something and assigning it to people answer different questions. When a common cause pushes both the choice and the result, the watched gap can be much bigger than the real effect. A coin that decides who gets what removes that push.
Before this lesson. L2-07, L2-04, L1-05
Grown-ups. The guide for adults is at the end of this page.
What you will learn¶
I can find a watched gap and an assigned gap from a table of made-up numbers.
I can explain why the two gaps differ when a common cause affects both the choice and the result.
I can explain why letting a coin decide gives a fair estimate of an effect.
Story: Does the study app work?¶
An imagined school, with made-up numbers, has 200 made-up students. Half of them, 100, are very motivated. A new study app is free to use. Of the 100 motivated students, 80 choose to use it. Of the other 100, only 20 do.
Leo looks at the test results. Among the 100 app users, 72 out of 100 pass. Among the 100 non-users, only 38 out of 100 pass. “That’s a gap of 34 points,” he says. “The app is amazing.”
Maya draws a causal graph: motivated -> uses app -> passes, and motivated -> passes. “Motivated students pass more anyway, and they also chose the app more. So the app users had a head start.” In this made-up model, the app adds just 10 points to anyone’s chance of passing. Motivation adds 40.
To test the app fairly, the school would let a coin decide who gets it. Then motivated and other students land in both groups, about half each. In the model, students given the app pass at about 60 out of 100. Students without it pass at about 50 out of 100. The gap is 10 points.
“Watching gave 34 points. Assigning gave 10 points,” says Leo. “Same students, same app.” Maya adds, “Watching asks who passes among people who chose the app. Assigning asks what happens if we give the app to people.” In symbols, P(pass | app) is 0.72, but P(pass | do(app)) is 0.60.
200 made-up students: first they choose the app, then a coin decides.
What the animation shows. A graph shows motivated -> uses app -> passes, plus motivated -> passes. Beside it, 200 made-up students appear as dots, violet if motivated and gray if not. First the students choose: rings appear around app users, mostly violet ones. App users pass at 72 out of 100, non-users at 38 out of 100, a gap of 34 points. Then scissors cut the arrow from motivated to uses app, and a coin points to uses app instead. The rings move so that exactly half of each color gets the app, the even split a fair coin gives on average. Now the app group passes at 60 out of 100 and the others at 50 out of 100, a gap of 10 points.
Hands-on: Watch, then assign¶
About 20 minutes.
You need:
10 cards numbered 0 to 9
Two coins
A pencil and paper for two tables
A calculator (optional)
The optional causal notebook, if you have a computer
Steps:
Shuffle the 10 cards. Each student you simulate needs a coin flip for motivation: heads means motivated, tails means not.
Watch round. Draw a card for app use. A motivated student uses the app on cards 0 to 7. Another student uses it only on cards 0 or 1.
Put the card back and shuffle. Draw again for passing. The student passes if the card is below their number: 8 for motivated with app, 7 for motivated without app.
For a student who is not motivated, the numbers are 4 with the app and 3 without. Simulate 20 students and record motivated, app, and pass.
Assign round. Flip the motivation coin as before. Then flip the second coin: heads gets the app, tails does not. Motivation plays no part.
Draw a card for passing with the same numbers. Simulate 20 more students and record them in a second table.
In each table, find the fraction who passed with the app and without it. Subtract to find each gap.
Compare your gaps with the model: 34 points for watching and 10 points for assigning. With only 20 students, expect a lot of wobble, so pool your results with a partner.
Practice together¶
Check the watched numbers in the made-up study. Of the 80 motivated app users, 80 percent pass. Of the 20 other app users, 40 percent pass. How many of the 100 app users pass?
Check your answer
80 percent of 80 is 64, and 40 percent of 20 is 8. So 64 + 8 = 72 of the 100 app users pass. That is 72 out of 100. Most app users were motivated, which pushed the number up.
Try it yourself¶
1. Now check the coin in the made-up study. It gives the app to 50 motivated students and 50 others. Motivated students with the app pass 80 percent of the time, others 40 percent. How many of the 100 pass?
Check your answer
80 percent of 50 is 40, and 40 percent of 50 is 20. So 40 + 20 = 60 of 100 pass, which is 60 out of 100. Compare 72 out of 100 when students chose.
2. An imagined news story says people who take vitamins are healthier, so vitamins work. Draw a graph with one possible common cause. What study would test the claim?
Check your answer
One common cause could be caring about health, which leads to taking vitamins and also to exercise and good food. A randomized trial, where chance decides who gets vitamins or a dummy pill, would estimate the effect.
Talk about it¶
Why does a coin flip make the two groups alike in motivation, at least on average?
Name a question where a randomized trial would be unfair or impossible. What could you do instead?
The watched gap was 34 points and the real effect in the model was 10 points. Could a watched gap ever be smaller than the real effect?
Check yourself¶
1. In the made-up app study, which gap estimates the effect of giving students the app?
A. The watched gap of 34 points
B. Neither, because only grades matter
C. The assigned gap of 10 points
Show the answer
C. The assigned gap of 10 points
When a coin decides, motivated students land about evenly in both groups. The remaining gap of about 10 points comes from the app.
2. Why was the watched gap so much bigger?
A. The coin flips were unlucky.
B. The app works better when you choose it.
C. Motivated students chose the app more and also passed more anyway.
Show the answer
C. Motivated students chose the app more and also passed more anyway.
Motivation is a common cause of using the app and of passing. Watching mixes its effect into the app’s gap.
3. Remember Lesson 5. In an imagined screening of 1,000 people, 1 percent have a condition. The test catches 90 percent of them, and 9 percent of the rest test positive. About how many positives have the condition? (Review of L2-05)
A. About 9 out of 10
B. About 9 out of 98
C. About 90 out of 100
Show the answer
B. About 9 out of 98
10 people have the condition and about 9 test positive. Of the other 990, about 89 test positive. So about 9 of the 98 positives have it.
Words to know¶
- intervene
- To set something on purpose for people, instead of letting them choose it.
- randomized trial
- A study where chance, like a coin flip, decides who gets each treatment.
- percentage points
- The gap between two percents. From 38 percent to 72 percent is a gap of 34 points.
For grown-ups: about this lesson
Why it matters. This lesson contrasts the observational quantity P(Y | X = 1) with the interventional quantity P(Y | do(X = 1)). Pearl’s overview defines the do-operator as replacing the mechanism that sets X, which in the graph removes the arrows into X Pearl, 2009. Hernan and Robins explain why randomization makes the treated and untreated groups exchangeable, so a randomized trial estimates the causal effect Hernan & Robins, 2020. OpenStax describes randomized experiments and the role of chance assignment Barbara Illowsky and Susan Dean, 2023. All rates come from the course’s tested model with motivation 50 percent, app use 80 or 20 percent, and pass chance 0.3 + 0.1 (app) + 0.4 (motivated). The watched gap is 34 points and the true effect is 10 points.
Watch for these mix-ups.
A big gap between people who chose a treatment and those who did not shows the treatment works. (A common cause can make the gap far bigger or smaller than the effect.)
Intervening means watching more carefully. (It means setting the value, which cuts the usual causes of the choice.)
A randomized trial gives the exact effect. (It estimates the effect; small trials wobble, which is why the activity results vary.)
To make it easier. Skip the card simulation. Work through the guided example and the first try-it with the 200 made-up students only.
To make it harder. Change the model so motivated students pick the app 95 percent of the time, and recompute the watched gap by hand. Or use the optional causal notebook to vary each setting and watch the two gaps.
Limits. All numbers are made up and come from a simple model in which the app adds the same 10 points for everyone. Real studies have dropouts, side effects and mixed-up groups, and real actions are not always as clean as the do-operator.
On the computer. Grown-ups can see where this idea leads later in Lab: Confounding and intervention, written for older learners.
Sources for this guide. Pearl, 2009Hernan & Robins, 2020Barbara Illowsky and Susan Dean, 2023
Next: L2-09 · Use vectors for two outcomes
- Pearl, J. (2009). Causal inference in statistics: An overview. Statistics Surveys, 3(none), 96–146. 10.1214/09-ss057
- Hernan, M., & Robins, J. (2020). Causal Inference: What If. https://miguelhernan.org/whatifbook
- Barbara Illowsky and Susan Dean. (2023). 1.4 Experimental Design and Ethics. OpenStax, Introductory Statistics 2e. https://openstax.org/books/introductory-statistics-2e/pages/1-4-experimental-design-and-ethics