11-james-traina-compound-science-skills-empirical-playbook

brycewang-stanford/Auto-Empirical-Research-SkillsResearchAudit pendingSnapshot 4194cdd910ac

Summary

This source did not publish a separate summary. Review SKILL.md before using the skill.

SKILL.md

Applied Micro Toolkit

Reference for applied micro research design: method selection, diagnostics, inference, pitfalls, reporting standards, and power analysis.

When to Use This Skill

Use when the user is:

Choosing between empirical methods for a causal question
Evaluating which identification strategy fits their data and setting
Running standard diagnostic tests and unsure which ones apply
Designing a study and needs to calculate statistical power
Reviewing or critiquing an empirical strategy
Preparing the "Empirical Strategy" section of a paper
Downloading macroeconomic or cross-national data (see references/data-sources.md for FRED/World Bank API access)

Skip when:

Implementation details for a specific method are needed (use causal-inference skill for IV, DiD, RDD, SC, matching)
The task is structural estimation (use structural-modeling skill)
The task is manuscript preparation or journal logistics (use submission-guide skill)
The task is formal identification proof (use identification-proofs skill)
The task is Bayesian model specification (use bayesian-estimation skill)

After selecting a method, the econometric-reviewer agent can review the implementation and the identification-critic agent can evaluate the identification argument.

Method Selection Decision Tree

Start with the fundamental question: What source of variation identifies the causal effect?

Step 1: What is your source of variation?

Source of Variation	Method Family	Key Assumption
Randomized assignment (with full compliance)

Is treatment timing staggered?
├── No → Classic 2x2 DiD (TWFE is fine)
└── Yes
    ├── Can treatment turn off (reversals)?
    │   ├── Yes → de Chaisemartin-D'Haultfoeuille (2020)
    │   └── No
    │       ├── Do you have never-treated units?
    │       │   ├── Yes → Callaway-Sant'Anna (2021) with never-treated controls
    │       │   └── No → Callaway-Sant'Anna with not-yet-treated controls
    │       │           or Sun-Abraham (2021)
    │       └── Are effects likely heterogeneous across cohorts?
    │           ├── Yes → Callaway-Sant'Anna or Sun-Abraham (NOT TWFE)
    │           └── No → TWFE is OK, but report Bacon decomposition

How many instruments for how many endogenous regressors?
├── Exactly identified (K instruments = K endogenous)
│   └── 2SLS (= IV = Wald estimator for single instrument)
├── Over-identified (K instruments > K endogenous)
│   ├── 2SLS (default)
│   ├── GMM (efficient, use if heteroskedasticity suspected)
│   └── LIML (less biased with weak instruments)
└── Under-identified (K instruments < K endogenous)
    └── Cannot identify all parameters — need more instruments or fewer endogenous regressors

Does crossing the threshold guarantee treatment?
├── Yes → Sharp RDD
└── No → Fuzzy RDD
    └── Is the running variable continuous?
        ├── Yes → Standard rdrobust
        └── No (discrete / few mass points)
            └── Cattaneo-Idrobo-Titiunik (2019) discrete RD methods

Is the selection-on-observables assumption plausible?
├── No → Need a different identification strategy
└── Yes
    ├── Do you need ATE or ATT?
    │   ├── ATE → IPW or AIPW
    │   └── ATT → Matching or IPW with ATT weights
    ├── Is the propensity score model well-specified?
    │   ├── Uncertain → Use AIPW (doubly robust)
    │   └── Confident → IPW or regression adjustment
    └── Many covariates or nonlinear confounding?
        ├── Yes → ML-based methods (causal forests, DML)
        └── No → Parametric PS model + AIPW

Method	Must-Run Diagnostics	Key Concern
IV / 2SLS	First-stage F (KP), reduced form, overid test	Weak instruments (F < 10), exclusion restriction
DiD (classic)	Pre-trend F-test, event study plot, raw means by group/period	Parallel trends violation
Staggered DiD	Bacon decomposition, Callaway-Sant'Anna group-time ATTs	Negative TWFE weights with heterogeneous effects
RDD	McCrary density test, covariate balance at cutoff, bandwidth sensitivity	Manipulation of running variable, extrapolation bias
Synthetic Control	Pre-fit RMSPE, permutation p-value, leave-one-out	Pre-period fit quality, donor pool sensitivity
Matching / AIPW	Overlap plots, Love plot (SMD before/after), Oster/Rosenbaum bounds	Lack of overlap, unobserved confounders
Structural	Convergence, identification rank condition, robustness to starting values	Global vs local optimum, identification failure

Mistake	Consequence	Fix
Clustering too fine (individual when treatment is at state level)	SEs too small; over-rejection	Cluster at the level of treatment assignment
Few clusters (< 30–40) with standard cluster-robust SEs	Poor finite-sample properties	Wild cluster bootstrap
Not clustering when treatment varies at group level	SEs dramatically understated	Always cluster at level of treatment assignment

Dimension	Design-Based	Model-Based
Source of randomness	Treatment assignment mechanism	Outcome draws from a superpopulation
Key assumption	Known or modeled treatment assignment	Correct outcome model specification
Examples	Experiments, RCTs, RDD, DiD, natural experiments	Structural models, matching, cross-sectional surveys
Advantages	Transparent; does not require outcome model	More powerful; extends to complex settings

MDE = 2.8 × sigma / sqrt(N)

Required N = (2.8 × sigma / MDE)²

Variable Type	Example	Why It Is Bad
Post-treatment outcome	Controlling for occupation when estimating returns to education	Education affects occupation; conditioning selects on an outcome of treatment
Mediator	Controlling for wages when estimating effect of training on employment	Blocks part of the causal effect
Collider	Conditioning on "survived" when estimating health effects	Opens a non-causal path

Mistake	Consequence	Fix
Running TWFE with staggered timing	Already-treated units used as controls; negative weights; estimate can have wrong sign	Use Callaway-Sant'Anna, Sun-Abraham, or other modern DiD estimator
Using single post-treatment indicator for all cohorts	Masks heterogeneity in treatment effects across cohorts	Estimate group-time ATTs separately, then aggregate
Not reporting the Bacon decomposition	Reader cannot assess how much of the TWFE estimate comes from problematic comparisons	Report `bacondecomp` output

11-james-traina-compound-science-skills-empirical-playbook

Summary

SKILL.md

Applied Micro Toolkit

When to Use This Skill

Method Selection Decision Tree

Step 1: What is your source of variation?

Step 2: Refinements Within Method Families

Standard Diagnostics by Method

Inference Frameworks

Clustering Decision Rule

Design-Based vs Model-Based Inference

Power Analysis

Research Design Checklist

Before Touching Data

During Analysis

Before Submission

Common Pitfalls

Bad Controls

Staggered DiD with Heterogeneous Effects

Forbidden Regressions

Integration

Related skills