Simpson's paradox in mix shift

Aggregates lie when mix shifts. Simpson's paradox: every segment gets worse, but the overall rate improves because more traffic comes from the high-converting segment — or the reverse. When a metric moves, ask: Did the rate change within segments, or did the segment mix change? Both can be true; they imply different actions (fix the product vs fix acquisition quality).

A blended coffee roast is a useful picture: add more of the strong beans and the cup tastes stronger even if both bags went stale. The analogy breaks when you have more than two segments, but the arithmetic is the same: overall rate is a weighted average of segment rates.

The big idea

Always segment before claiming a funnel "improved." Typical cuts: channel, platform, new vs returning, plan, country. The same habit applies to retention curves — a blend can fall because you bought more paid traffic, not because the product got worse.

Take any product claim in Slack ("retention is up") and ask for cohort, window and segment. That habit alone will make you the person others trust with ambiguous data.

Walking through it

Two channels, two weeks.

Week 1

Week 2 (each channel worse, mix shifts toward Search)

Here overall fell slightly. Flip the mix — keep week-2 per-channel rates but reverse volumes:

Search: 2,000 visits × 4.5% = 90 paid
Social: 8,000 visits × 0.8% = 64 paid
Total:  10,000 visits, 154 paid → 1.54% overall

Compare to week 1's 4.2%: overall collapsed because of mix, while you might have been told "Search is still our best channel at 4.5%."

Now the dangerous case — overall up while both channels down:

Week A: Search 1,000 @ 10% (100 paid), Social 9,000 @ 2% (180) → total 280 / 10,000 = 2.8%

Week B: Search 8,000 @ 9% (720), Social 2,000 @ 1.5% (30) → total 750 / 10,000 = 7.5%

Overall conversion nearly tripled (2.8% → 7.5%) while Search fell 10%→9% and Social fell 2%→1.5%. Celebrating the aggregate would ship the wrong roadmap.

Common mistakes

• Celebrating the blended rate after a campaign changed channel mix.
• Fixing the product when the real change was acquisition quality — or the reverse.
• One segment cut. Mix can hide in platform, geo or new vs returning at the same time.

Check yourself

1. Week A overall 2.8%, week B overall 7.5%, both channels down. What happened?

Mix shifted toward Search (the better channel). Weighted average rose while every segment worsened.

2. A growth channel scale-up lands. Blended activation falls. What two questions first?

Did activation fall inside each channel, or did mix shift toward a worse channel? Those imply product work vs targeting work.

Key takeaways

• Overall rate is a weighted average. Mix shift can move it without any segment improving.
• Segment before celebrating aggregates — this is common in growth data.
• Product vs acquisition diagnosis depends on whether within-segment rates moved.

Further learning

Mixpanel on Simpson's paradox and why segmentation is not optional.

Simpson's paradox and segmenting data (Mixpanel)Avoid misleading insights! Learn how Simpson’s paradox affects data analysis and why segmenting data is crucial for accurate businessMixpanel

Next: the SQL shapes behind these questions in analytics SQL.