Building Paper Screening Rubrics

Overview

Core principle: Build screening rubrics collaboratively through brainstorming → test → refine → automate → review → iterate.

Good rubrics come from understanding edge cases upfront and testing on real papers before bulk screening.

When to Use

Use this skill when:

• Starting a new literature search that will screen 50+ papers
• Current rubric misclassifies papers (false positives/negatives)
• Need to define "relevance" criteria before automated screening
• Want to update criteria and re-screen cached papers
• Building helper scripts for evaluating-paper-relevance

When NOT to use:

• Small searches (<20 papers) - manual screening is fine
• Rubric already works well - no need to rebuild
• One-off exploratory searches

Two-Phase Process

Phase 1: Collaborative Rubric Design

Step 1: Brainstorm Relevance Criteria

Ask domain-agnostic questions to understand what makes papers relevant:

Core Concepts:

• "What are the key terms/concepts for your research question?"
  • Examples: specific genes, proteins, compounds, diseases, methods, organisms, theories
• "Are there synonyms or alternative names?"
• "Any terms that should EXCLUDE papers (false positives)?"

Data Types & Artifacts:

• "What type of information makes a paper valuable?"
  • Quantitative measurements (IC50, expression levels, population sizes, etc.)
  • Protocols or methods
  • Datasets with accessions (GEO, SRA, PDB, etc.)
  • Code or software
  • Chemical structures
  • Sequences or genomes
  • Theoretical models