A unified toolkit for discovering, retrieving metadata, and downloading academic papers
across all major sources. The right API depends on the task — this skill routes to the best source.
Decision Tree: Which API to Use
What is needed?
│
├─ SEARCH for papers on a topic
│ ├─ arXiv preprints (CS/AI/Physics/Econ) ──→ arXiv API
│ ├─ NBER working papers (economics) ───────→ NBER API
│ ├─ SSRN working papers (social science) ──→ OpenAlex (primary) + CrossRef prefix:10.2139
│ ├─ Published journal articles ────────────→ CrossRef API or OpenAlex
│ └─ Broad multi-source search ─────────────→ OpenAlex API
│
├─ GET METADATA for a known paper
│ ├─ Have DOI ──→ CrossRef (structured) or OpenAlex (richer)
│ ├─ Have arXiv ID ──→ arXiv API
│ ├─ Have NBER number ──→ NBER API
│ └─ Have SSRN abstract ID ──→ OpenAlex DOI lookup (DOI = 10.2139/ssrn.{id})
│
├─ FIND/DOWNLOAD PDF
│ ├─ Have DOI ──→ Unpaywall API (check OA) → OpenAlex (alt PDF links)
│ ├─ arXiv paper ──→ Direct: https://arxiv.org/pdf/{id}
│ ├─ NBER paper ──→ Direct: https://www.nber.org/system/files/working_papers/w{n}/w{n}.pdf
│ └─ SSRN paper ──→ Unpaywall (~30%) or institutional repos (EconStor, IZA)
│
└─ IMPORT TO ZOTERO
└─ Always use zotero_add_by_doi (best metadata + auto PDF via Unpaywall cascade)
1. OpenAlex API (Best All-Round)
Free, no API key needed. Covers 250M+ works including SSRN, NBER, arXiv, all journals.
Add [email protected] for polite pool (faster responses).
Search by Source
# SSRN papers (source ID: S4210172589, NOT S4306400806 which is Europe PMC)
curl -s "https://api.openalex.org/works?filter=primary_location.source.id:S4210172589,default.search:social+preferences,publication_year:2024-2026&per_page=25&[email protected]"
# NBER papers
curl -s "https://api.openalex.org/works?filter=primary_location.source.id:S4210174556,default.search:behavioral+economics&per_page=25&[email protected]"
import requests
def openalex_search(query, source_id=None, year_from=2024, year_to=2026, per_page=25):
"""Search OpenAlex. source_id: S4210172589 (SSRN), S4210174556 (NBER), etc."""
filters = [f"default.search:{query}", f"publication_year:{year_from}-{year_to}"]
if source_id:
filters.insert(0, f"primary_location.source.id:{source_id}")
params = {
"filter": ",".join(filters),
"per_page": per_page,
"mailto": "[email protected]"
}
data = requests.get("https://api.openalex.org/works", params=params).json()
results = []
for w in data.get("results", []):
pdf_urls = [loc["pdf_url"] for loc in w.get("locations", []) if loc.get("pdf_url")]
results.append({
"title": w["title"],
"doi": w.get("doi", "").replace("https://doi.org/", ""),
"year": w.get("publication_year"),
"authors": [a["author"]["display_name"] for a in w.get("authorships", [])],
"cited_by": w.get("cited_by_count", 0),
"pdf_urls": pdf_urls,
"landing_url": (w.get("primary_location") or {}).get("landing_page_url", ""),
})
return {"total": data.get("meta", {}).get("count", 0), "results": results}
Key Source IDs
Full reference: See references/journal_identifiers.md for a comprehensive table of 77 verified
OpenAlex Source IDs and CrossRef ISSNs covering Top 5 economics, AEA journals, finance, management,
marketing, accounting, IS, and 15+ field journal categories.
Source
OpenAlex ID
CrossRef ISSN
SSRN Electronic Journal
S4210172589
1556-5068
NBER Working Papers
S2809516038
N/A
American Economic Review
S23254222
0002-8282
Quarterly Journal of Economics
S203860005
0033-5533
Econometrica
S95464858
0012-9682
Warning: OpenAlex source IDs are opaque and not guessable. Always verify
via https://api.openalex.org/sources?search=journal+name before using a new ID.
For verified IDs of 77 journals, consult references/journal_identifiers.md.
PDF from OpenAlex
primary_location.pdf_url is null for SSRN papers (Elsevier blocks it).
Check locations[].pdf_url for third-party repos — ~24% of SSRN papers have PDFs
via EconStor, IZA, MPRA, etc. These are real, downloadable PDFs.
2. CrossRef API (Best for DOI Metadata)
Free, no key needed. Add mailto= for 50 req/sec (vs 1 req/sec without).
Search with SSRN DOI Prefix
# All SSRN papers matching a query
curl -s "https://api.crossref.org/works?query=%22social+preferences%22&filter=prefix:10.2139,from-pub-date:2024-01-01&rows=25&sort=relevance&[email protected]"
These predictable DOI formats allow constructing DOIs from paper IDs
without making extra API calls.
Limitations
CrossRef returns title, authors, DOI, date, journal — but no abstracts for SSRN papers
(Elsevier doesn't submit them to CrossRef). Use OpenAlex for richer metadata.
3. arXiv API (Preprints — CS/AI/Physics/Math/Econ)
Free, no key needed. Returns Atom XML. Rate limit: ~1 request per 3 seconds.
Caveat: The cat: filter in search_query does not always reliably restrict results
to specified categories — arXiv may still return papers from other fields.
Verify category matches in results and filter client-side if needed.
Direct PDF
https://arxiv.org/pdf/{id} # e.g. https://arxiv.org/pdf/2401.12345
https://arxiv.org/abs/{id} # abstract page
4. NBER API (Economics Working Papers)
Free, no key needed. Returns JSON.
# Search
curl -s "https://www.nber.org/api/v1/working_page_listing/contentType/working_paper/_/_/search?page=1&perPage=20&q=social+preferences"
# Filter by Program (replace _/_ with programs/{name})
curl -s "https://www.nber.org/api/v1/working_page_listing/contentType/working_paper/programs/Labor+Studies/search?page=1&perPage=20&q=minimum+wage"
# New this week
curl -s "https://www.nber.org/api/v1/working_page_listing/contentType/working_paper/_/_/search?page=1&perPage=20&newThisWeek=true"
Programs
Labor Studies, Economic Fluctuations and Growth, Industrial Organization,
Public Economics, Development Economics, International Finance and Macroeconomics,
Corporate Finance, Asset Pricing, Health Economics, etc.
Response Processing
Authors field contains HTML <a> tags — strip with: re.sub(r'<[^>]+>', '', text)
No built-in date filter — API sorts by relevance. Filter client-side by displaydate.
Free, no key needed. Email as query param. 100,000 requests/day.
curl -s "https://api.unpaywall.org/v2/10.2139/[email protected]" \
| python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f'OA: {d[\"is_oa\"]} ({d[\"oa_status\"]})')
pdf = (d.get('best_oa_location') or {}).get('url_for_pdf')
print(f'PDF: {pdf or \"none\"}')
# Check all locations for alternative PDFs
for loc in d.get('oa_locations', []):
if loc.get('url_for_pdf'):
print(f'Alt PDF: {loc[\"url_for_pdf\"]} ({loc[\"host_type\"]})')
"
OA Status Values
Status
Meaning
gold
Published OA by publisher
green
Free copy in a repository
hybrid
OA article in subscription journal
bronze
Free to read on publisher site
closed
No free version found
Platform-Specific PDF Availability
Source
Unpaywall PDF Rate
Why
arXiv
~100%
Native OA, direct PDF links
NBER
~97%
Green OA via author/institution repos
SSRN
~30%
Cloudflare blocks Unpaywall; only finds institutional repo mirrors
For SSRN papers where Unpaywall returns url_for_pdf=null, check oa_locations[]
for institutional repositories (EconStor, university repos) — follow their landing pages
to find PDF links.
6. Semantic Scholar API (Citations & Impact)
Free, no key needed. Rate limit: 100 requests per 5 minutes.
Warning: As of 2026, Semantic Scholar may return 429 errors frequently.
Consider registering for an API key at https://www.semanticscholar.org/product/api
for higher limits. Add header: x-api-key: YOUR_KEY.
