54-scdenney-open-science-skills-skills-list-experiment

brycewang-stanford/Auto-Empirical-Research-SkillsResearchAudit pendingSnapshot dac729da20a3

Summary

This source did not publish a separate summary. Review SKILL.md before using the skill.

SKILL.md

List Experiment Designer

Related skills: Use alongside hypothesis-building (state π and a SESOI before design choices), survey-design (mode effects, question ordering, and pre-testing of control items), and methods-reporting (deposit list wording, randomization seed, list package version, and ict.test / ict.hausman.test / ictreg() output).

Instructions

1. Pre-Design: Is a List Experiment Warranted?

Assess sensitivity bias first: Before committing to a list experiment, consult domain-specific evidence on sensitivity bias. Blair, Coppock, and Moor's (2020) meta-analysis of 30 years of list experiments shows that sensitivity biases are typically smaller than 10 percentage points. A list experiment is not automatically the right choice for any sensitive topic.
Social reference theory: Sensitivity bias is largest when (a) the social norm on the topic is strong, (b) the norm is clear and widely shared, and (c) respondents believe others can infer their true attitude from their response (Blair et al. 2020). Evaluate all three conditions before deciding.
Precision cost: List experiments require approximately 10 times more respondents than a direct question to achieve equivalent precision. The trade-off is only favorable when the expected sensitivity bias exceeds the precision loss (Blair et al. 2020). If the topic is sensitive but the expected bias is small (< 5pp), a direct question with neutral framing is often preferable.
Empirical benchmarks by domain: Voter turnout (~5–15pp overreport, wide confidence intervals), clientelism and vote-buying (~5–15pp underreport), racial prejudice (near-zero sensitivity bias — Blair et al. 2020 find little evidence respondents conceal prejudice on direct questions), authoritarian regime support (highly context-dependent and often dominated by artificial deflation rather than preference falsification). Use these as priors when no domain-specific estimates exist.

2. Basic Design

The floor/ceiling constraint: Select control items so that virtually no respondent would endorse all N control items (ceiling) or zero control items (floor) when assigned to the treatment group. A respondent at the ceiling cannot truthfully report N+1 even if they hold the sensitive attitude; one at the floor cannot hide a "1" count. Both violations bias estimates downward (artificial deflation) and compromise identification (Blair & Imai 2012).
Target prevalence range for control items: Each control item should have expected prevalence between 20% and 80% in the population. The sum of expected control item endorsements should have low variance — ideally each respondent endorses roughly 1–3 of N control items.
Item independence: Control items should be uncorrelated with the sensitive item. If control items tap dimensions that predict the sensitive attitude, the no-design-effect assumption is threatened.
Conventional vs. placebo vs. mixed control list design: Three design options exist for managing measurement error from inattentive respondents (Agerberg & Tannenberg 2021):
- Conventional: Standard baseline items, no placebo. Biased if nonstrategic error inflates or deflates counts.
- Placebo: Replace one control item with a universally false statement (all should answer "no") to equate list length in treatment and control. Riambau & Ostwald (2021) show this reduces mechanical inflation — the tendency for treatment group respondents to report more true items simply due to list length, especially among low-education respondents. However, Agerberg & Tannenberg (2021) show placebo items do not universally reduce bias and can increase it under some conditions.
- Mixed control list: Combines conventional and placebo items. Preferred when the expected direction and magnitude of sensitivity bias is uncertain.
- Choice rule: When mechanical inflation is the primary concern (inattentive or acquiescent respondents inflating counts), include a placebo item. When artificial deflation is the primary concern (ceiling effects), use a conventional design and select items with lower prevalence. When uncertain, use the mixed control list.

54-scdenney-open-science-skills-skills-list-experiment

Summary

SKILL.md

List Experiment Designer

Instructions

1. Pre-Design: Is a List Experiment Warranted?

2. Basic Design

3. Control List Design

4. Design Variants

5. Estimators

6. Identifying Assumptions and Diagnostics

7. Common Problems and Corrections

8. Statistical Power

Quality Checks

Example

Related skills