Evaluating Paper Relevance

brycewang-stanford/Auto-Empirical-Research-SkillsResearchAudit passedSnapshot 476138a317b3

Installs: 0

Summary

Two-stage paper screening - abstract scoring then deep dive for specific data extraction

SKILL.md

Evaluating Paper Relevance

Overview

Two-stage screening process: quick abstract scoring followed by deep dive into promising papers.

Core principle: Precision over breadth. Find papers that actually contain the specific data/methods user needs, not just topically related papers.

When to Use

Use this skill when:

Have list of papers from search
Need to determine which papers have relevant data
User asks for specific information (measurements, protocols, datasets, etc.)
Screening papers one-by-one
Any research domain (medicinal chemistry, genomics, ecology, computational methods, etc.)

Choosing Your Approach

Small searches (<50 papers):

Manual screening with progress reporting
Use papers-reviewed.json + SUMMARY.md only
No helper scripts needed
Report progress to user for every paper

Large searches (50-150 papers):

Consider helper scripts (screen_papers.py + deep_dive_papers.py)
Use Progressive Enhancement Pattern (see Helper Scripts section)
Create README.md with methodology
May want TOP_PRIORITY_PAPERS.md for quick reference
Use richer JSON structure (evaluated-papers.json categorized by relevance)
Consider using subagent-driven-review skill for parallel screening

Very large searches (>150 papers):

Definitely use helper scripts with Progressive Enhancement Pattern
Create full auxiliary documentation suite (README.md, TOP_PRIORITY_PAPERS.md)
Consider citation network analysis
Plan for multi-week timeline
Strongly consider subagent-driven-review skill for parallelization
May need multiple consolidation checkpoints

Two-Stage Process

Stage 1: Abstract Screening (Fast)

📄 [N/Total] Screening: "Paper Title"
   Abstract score: 8 → Fetching full text...

📄 [N/Total] Screening: "Paper Title"
   Abstract score: 4 → Skipping (insufficient relevance)

curl -s "https://www.ebi.ac.uk/chembl/api/data/document.json?doi=$doi"

# Check if available in PMC
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pmc&term=PMID[PMID]&retmode=json"

# If found, fetch full text XML via API
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=PMCID&rettype=full&retmode=xml"

# Or fetch HTML directly (note: use pmc.ncbi.nlm.nih.gov, not www.ncbi.nlm.nih.gov/pmc)
curl "https://pmc.ncbi.nlm.nih.gov/articles/PMCID/"

# Try publisher link
curl -L "https://doi.org/10.1234/example.2023"
# May hit paywall - check response

curl "https://api.unpaywall.org/v2/DOI?email=USER_EMAIL"
# Often finds versions in repositories, preprint servers, author copies
# IMPORTANT: Ask user for their email if not already provided - do NOT use [email protected]

⚠️  Paper behind paywall, checking Unpaywall...
✓ Found open access version at [repository/preprint server]

⚠️  Paper behind paywall, checking Unpaywall...
✗ No open access version available - continuing with abstract only

# Examples for different domains
grep -i "IC50\|Ki\|MIC" paper.xml                    # Medicinal chemistry
grep -i "expression\|FPKM\|RNA-seq" paper.xml        # Genomics
grep -i "abundance\|population\|sampling" paper.xml  # Ecology
grep -i "algorithm\|github\|code" paper.xml          # Computational

{
  "doi": "10.1234/medchem.2023",
  "title": "Novel kinase inhibitors...",
  "relevance_score": 9,
  "findings": {
    "data_found": [
      "IC50 values for compounds 1-12 (Table 2)",
      "Selectivity data (Figure 3)",
      "Synthesis route (Scheme 1)"
    ],
    "key_results": [
      "Compound 7: IC50 = 12 nM",
      "10-step synthesis, 34% yield"
    ]
  }
}

{
  "doi": "10.1234/genomics.2023",
  "title": "Gene expression in disease...",
  "relevance_score": 8,
  "findings": {
    "data_found": [
      "RNA-seq data for 50 samples (GEO: GSE12345)",
      "Differential expression results (Table 1)",
      "Gene set enrichment analysis (Figure 4)"
    ],
    "key_results": [
      "123 genes upregulated (FDR < 0.05)",
      "Pathway enrichment: immune response"
    ]
  }
}

{
  "doi": "10.1234/compbio.2023",
  "title": "Novel alignment algorithm...",
  "relevance_score": 9,
  "findings": {
    "data_found": [
      "Algorithm pseudocode (Methods)",
      "Code repository (github.com/user/tool)",
      "Benchmark results (Table 2)"
    ],
    "key_results": [
      "10x faster than BLAST",
      "98% accuracy on test dataset"
    ]
  }
}

# If PDF available
curl -L -o "papers/$(echo $doi | tr '/' '_').pdf" "https://doi.org/$doi"

# Download SI files if URLs found
curl -o "papers/${doi}_supp.zip" "https://publisher.com/supp/file.zip"

{
  "10.1234/example.2023": {
    "pmid": "12345678",
    "status": "relevant",
    "score": 9,
    "source": "pubmed_search",
    "timestamp": "2025-10-11T10:30:00Z",
    "found_data": ["IC50 values", "synthesis methods"],
    "has_full_text": true,
    "chembl_id": "CHEMBL1234567"
  }
}

{
  "10.1234/another.2023": {
    "pmid": "12345679",
    "status": "not_relevant",
    "score": 4,
    "source": "pubmed_search",
    "timestamp": "2025-10-11T10:31:00Z",
    "reason": "no activity data, review paper"
  }
}

### [Novel kinase inhibitors with improved selectivity](https://doi.org/10.1234/medchem.2023) (Score: 9)

**DOI:** [10.1234/medchem.2023](https://doi.org/10.1234/medchem.2023)
**PMID:** [12345678](https://pubmed.ncbi.nlm.nih.gov/12345678/)
**ChEMBL:** [CHEMBL1234567](https://www.ebi.ac.uk/chembl/document_report_card/CHEMBL1234567/)

**Key Findings:**
- IC50 values for 12 inhibitors (Table 2)
- Compound 7: IC50 = 12 nM, >80-fold selectivity
- Synthesis route (Scheme 1, page 4)

**Files:** PDF, supplementary data

### [Transcriptomic analysis of disease progression](https://doi.org/10.1234/genomics.2023) (Score: 8)

**DOI:** [10.1234/genomics.2023](https://doi.org/10.1234/genomics.2023)
**PMID:** [23456789](https://pubmed.ncbi.nlm.nih.gov/23456789/)
**Data:** [GEO: GSE12345](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE12345)

**Key Findings:**
- RNA-seq data: 50 samples, 3 conditions
- 123 differentially expressed genes (FDR < 0.05)
- Immune pathway enrichment (Figure 3)

**Files:** PDF, supplementary tables with gene lists

### [Fast sequence alignment with novel algorithm](https://doi.org/10.1234/compbio.2023) (Score: 9)

**DOI:** [10.1234/compbio.2023](https://doi.org/10.1234/compbio.2023)
**Code:** [github.com/user/tool](https://github.com/user/tool)

**Key Findings:**
- New alignment algorithm (pseudocode in Methods)
- 10x faster than BLAST, 98% accuracy
- Benchmark datasets available

**Files:** PDF, code repository linked

📄 [15/127] Screening: "Selective BTK inhibitors..."
   Abstract score: 8 → Fetching full text...
   ✓ Found IC50 data for 8 compounds (Table 2)
   ✓ Selectivity data vs 50 kinases (Figure 3)
   → Added to SUMMARY.md

📄 [23/89] Screening: "Gene expression in liver disease..."
   Abstract score: 9 → Fetching full text...
   ✓ RNA-seq data available (GEO: GSE12345)
   ✓ 123 DEGs identified (Table 1, FDR < 0.05)
   → Added to SUMMARY.md

📄 [7/45] Screening: "Novel phylogenetic algorithm..."
   Abstract score: 8 → Fetching full text...
   ✓ Code available (github.com/user/tool)
   ✓ Benchmark results (10x faster, Table 2)
   → Added to SUMMARY.md

📊 Progress: Reviewed 30/127 papers
   - Highly relevant: 3
   - Relevant: 5
   - Currently screening paper 31...

Score	Meaning	Action
0-4	Not relevant	Skip, brief note in summary
5-6	Possibly relevant	Note for later, skip deep dive for now
7-8	Relevant	Deep dive, extract data, add to summary
9-10	Highly relevant	Deep dive, extract data, follow citations, highlight in summary

# research-sessions/YYYY-MM-DD-query/screen_papers.py

Task	Action
Check if reviewed	Look up DOI in papers-reviewed.json
Score abstract	Keywords (0-3) + Data type (0-4) + Specificity (0-3)
Get full text	Try PMC → DOI → Unpaywall → Preprints
Find data	Grep for terms, focus on Methods/Results/Tables
Download PDF	`curl -L -o papers/FILE.pdf URL`
Update tracking	Add to papers-reviewed.json + SUMMARY.md

# Top Priority Papers

## Tier 1: Must-Read (Score 10)

### [Paper Title](https://doi.org/10.xxxx/yyyy) (Score: 10)

**DOI:** [10.xxxx/yyyy](https://doi.org/10.xxxx/yyyy)
**PMID:** [12345678](https://pubmed.ncbi.nlm.nih.gov/12345678/)
**Full text:** ✓ PMC12345678

**Key Findings:**
- Finding 1
- Finding 2

---

## Tier 2: High-Value (Score 8-9)

[Additional papers organized by priority...]

Evaluating Paper Relevance

Summary

SKILL.md

Evaluating Paper Relevance

Overview

When to Use

Choosing Your Approach

Two-Stage Process

Stage 1: Abstract Screening (Fast)

Stage 2: Deep Dive (Thorough)

1. Check ChEMBL (for medicinal chemistry papers)

2. Fetch Full Text

2. Scan for Relevant Content

3. Extract Findings

4. Download Materials

5. Update Tracking Files

Progress Reporting

Integration with Other Skills

Scoring Rubric

Helper Scripts (Optional)

Progressive Enhancement Pattern (Recommended for 50+ papers)

Common Mistakes

Quick Reference

Next Steps

Auxiliary Files (for large searches >100 papers)

README.md Template

TOP_PRIORITY_PAPERS.md Template

Related skills