ref_hsieh2016

Up: presentation-increasing-response-rates-incentives

You Get Who You Pay for: The Impact of Incentives on Participation Bias

Reading: Hsieh, Gary, and Rafał Kocielnik. 2016. “You Get Who You Pay for: The Impact of Incentives on Participation Bias.” In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, San Francisco California USA: ACM, 823–35. doi:10.1145/2818048.2819936.

Overview

This study explores how different rewards affect who joins a crowdsourcing task on Amazon Mechanical Turk and how that changes the task results.
They tested three rewards—fixed cash, a lottery, and a charity donation—in two experiments with a brainstorming task.
They found that
- lotteries attract risk-takers and people who value new experiences (openness-to-change)
- charity rewards draw those who care about others (self-transcendence).

What They Did

Why They Did It: Rewards are used to get people to join tasks like surveys or crowdsourcing, but most studies focus on how many participate, not who. Different rewards might attract different types of people, affecting task results. They wanted to see how rewards create “participation bias” (who joins) and impact outcomes.
The Plan:
- Who: Workers on Mechanical Turk, an online platform for small paid tasks.
- Task: A brainstorming activity (Experiment 1: creative uses for a quarter; Experiment 2: uses for a brick). Participants had to list at least 5 ideas but could add up to 10.
- Rewards: Three types, all post-paid (given after task completion):
  - Fixed: A guaranteed small payment (e.g., $0.25).
  - Lottery: A chance to win a bigger prize (e.g., $25 with 1/100 odds, same expected value as fixed).
  - Charity: A donation (e.g., $0.25) to a charity like the Red Cross instead of personal payment.
- What They Checked:
  - Who Joins: Do rewards attract people with specific traits, like risk-taking or values (using Schwartz’s Basic Human Values: openness-to-change for new experiences, self-transcendence for caring about others)?
  - Task Results: Do different groups produce more ideas, more original ideas, or more diverse ideas?

Experiment

Experiment 1 (165 participants):
- Setup: Participants first did a $1 s u r v eyo n v a l u es an d p erso na l i t y (8 min u t es) . T h e n, t h ey w ere in v i t e d t o t h eo pt i o na l$ 0.25 brainstorming task. Half chose between lottery ( $25, 1/100 c han ce) or f i x e d ($ 0.25); half chose between charity ( $0.25 d o na t i o n) or f i x e d ($ 0.25).
- Goal: See which rewards people pick and how that affects task performance.
- Participants: 83 in lottery vs. fixed (36% picked lottery); 82 in charity vs. fixed (33% picked charity).
Experiment 2 (809 participants after exclusions):
- Setup: Same initial $1 s u r v ey, b u tp a r t i c i p an t s w ere a ss i g n e d o n eo f se v e n co n d i t i o n s f or t h e b r ain s t or min g t a s k : n ore w a r d ($ 0), fixed ( $0.05,$ 0.25), lottery ( $5 or$ 25, 1/100 chance), or charity ( $0.05,$ 0.25 donation). They chose to participate or not.
- Goal: Test if rewards attract different people when offered directly (not a choice between two).
- Participants: 47% opted out; participation varied (e.g., 65% for $0.25 f i x e d, 42$ 0.25 charity).

What Happened

Who Joined (Participation Bias):
- Lottery Reward:
  - Experiment 1: Attracted people who were more risk-seeking (1.99 vs. 1.48 on a 5-point scale, p < 0.001) and valued openness-to-change (0.35 vs. -0.05, p < 0.05), especially stimulation (liking risk and adventure). They were 8.3 times more likely to pick lottery per unit of risk-seeking (p < 0.01) and 2.3 times per unit of openness-to-change (p < 0.05).
  - Experiment 2: Attracted those with openness-to-change values (1.40 times more likely to join, p < 0.10), but risk-seeking wasn’t significant (possibly because there was no direct comparison to fixed pay).
- Charity Reward:
  - Experiment 1: Attracted those with stronger self-transcendence values (1.06 vs. 0.44, p < 0.001), who were 3 times more likely to pick charity per unit of self-transcendence (p < 0.01). They were also older (42 vs. 34 years, p < 0.05) and more often female (79% vs. 53%, p < 0.05).
  - Experiment 2: Attracted those with self-transcendence values (1.68 times more likely to join, p < 0.05). Older people were slightly more likely to join (p < 0.05).
- Fixed Reward:
  - Attracted those with self-enhancement values (focus on personal gain) in Experiment 1 (p < 0.001). In Experiment 2, $0.25 fixed had the highest participation (65%), but no strong value bias except slightly favoring self-direction.
- No Reward:
  - In Experiment 2, attracted those with openness-to-change values (1.58 times more likely, p < 0.05), likely because they enjoyed the creative task itself.
Task Results:
- Experiment 1:
  - Lottery Group: Generated more ideas (6.53 vs. 5.70, p < 0.05), more original ideas (2.90 vs. 2.63 on a 5-point scale, p < 0.01), and more diverse ideas (4.61 vs. 4.02 categories, p < 0.05). They added ~2 times more extra ideas (p < 0.05), driven by self-direction values.
  - Charity Group: No difference in ideas generated (6.37 vs. 6.13), originality (2.74 vs. 2.72), or feasibility (4.30 vs. 4.31) compared to fixed.
- Experiment 2:
  - Lottery ($25): Generated 1.2 times more extra ideas than no-reward baseline (not significant, p = 0.18), driven by self-direction and stimulation values.
  - Fixed ( $0.25) an d C ha r i t y ($ 0.25): Generated fewer extra ideas (0.63 and 0.50 times baseline, p < 0.01), suggesting less intrinsic interest.
  - Low Rewards ( $0.05,$ 5 lottery, $0.05 charity): No difference from baseline in ideas or quality.
  - Originality/Feasibility: No significant differences across conditions (unlike Experiment 1).
Why People Chose Rewards:
- Lottery: Liked gambling or thought odds were good (e.g., “I’d rather take a chance on $25 than 0.25 cents”).
- Fixed: Wanted guaranteed money or disliked gambling (e.g., “I hate raffles”).
- Charity: Wanted to help others (e.g., “It’s good to give”).
- Opted Out (Experiment 2): Cited low pay, no time, or disliking brainstorming.

What It Means

Key Points:
- Rewards shape who joins a task:
  - Lotteries attract risk-takers and those who like new experiences (openness-to-change, especially stimulation).
  - Charity rewards draw those who value helping others (self-transcendence).
  - Fixed rewards appeal to those focused on personal gain (self-enhancement).
- This “participation bias” affects results:
  - Lottery groups generated more and more original ideas (especially in Experiment 1) because openness-to-change (self-direction) correlates with creativity.
  - Charity and fixed groups didn’t outperform no-reward baseline in creativity.
- Higher rewards ( $0.25) ha d s t ro n g ere ff ec t s t han l o w o n es ($ 0.05).
Why It Worked:
- People’s values (per Schwartz’s model) guide their choices. Lottery appeals to thrill-seekers; charity to altruists.
- Creative tasks suit those with openness-to-change, so lotteries boosted output by attracting them.
- Per leverage-salience theory, rewards make the task more appealing to those who value them.
Why Some Results Differed:
- Experiment 1’s choice (lottery vs. fixed) highlighted risk, boosting risk-seeking’s role. Experiment 2’s single reward weakened this.
- Originality differences in Experiment 1 didn’t repeat in Experiment 2, possibly due to task differences (quarter vs. brick) or coding variations.
Tips for Crowdsourcing:
- Use lotteries for creative tasks to attract idea-generators.
- Use charity rewards to attract altruistic participants for tasks needing diverse perspectives.
- Offer multiple reward types to get a diverse participant pool and avoid bias.
- Tailor rewards to target specific groups (e.g., charity for self-transcendent people).
- Higher rewards work better than tiny ones.

Quartz 4

Explorer