Academic Web Scraping Guide

Overview

Research often requires collecting data from the web -- whether it is bibliographic metadata from academic databases, experimental datasets from public repositories, social media posts for computational social science, or economic indicators from government portals. Web scraping and API-based data collection are essential skills for modern researchers across disciplines.

This guide covers both approaches: structured API access for platforms that provide one, and web scraping for when no API exists. It emphasizes ethical data collection practices, including respecting robots.txt, rate limiting, terms of service compliance, and IRB considerations for human-subject data. The goal is to collect research data reliably and responsibly.

Whether you are building a dataset for a machine learning paper, collecting metadata for a systematic review, or gathering public data for policy research, these patterns help you do it correctly and efficiently.

API-Based Data Collection

APIs are always preferable to scraping when available. They provide structured data, are officially supported, and have clear usage terms.

Academic APIs

API	Data	Rate Limit	Auth
OpenAlex	Papers, authors, venues, concepts	100K req/day	Email in header
Crossref	DOI metadata	50 req/sec (polite pool)	Email in header
PubMed (Entrez)	Biomedical literature	10 req/sec (with key)	API key (free)
arXiv	Preprints	1 req/3sec	None
CORE	Open access papers	10 req/sec	API key (free)

Example: Collecting Papers from OpenAlex

import requests
import time

class OpenAlexClient:
    BASE_URL = "https://api.openalex.org"

    def __init__(self, email):
        self.session = requests.Session()
        self.session.headers.update({
            'User-Agent': f'ResearchBot/1.0 (mailto:{email})'
        })

    def search_works(self, query, filters=None, per_page=25, max_results=100):
        """Search for works with optional filters."""
        results = []
        page = 1

        while len(results) < max_results:
            params = {
                'search': query,
                'per_page': min(per_page, max_results - len(results)),
                'page': page,
            }
            if filters:
                params['filter'] = ','.join(f'{k}:{v}' for k, v in filters.items())

            resp = self.session.get(f'{self.BASE_URL}/works', params=params)
            resp.raise_for_status()
            data = resp.json()

            works = data.get('results', [])
            if not works:
                break

            results.extend(works)
            page += 1
            time.sleep(0.1)  # Polite rate limiting

        return results[:max_results]

    def get_work(self, openalex_id):
        """Get a single work by OpenAlex ID."""
        resp = self.session.get(f'{self.BASE_URL}/works/{openalex_id}')
        resp.raise_for_status()
        return resp.json()

# Usage
client = OpenAlexClient(email="[email protected]")
papers = client.search_works(
    "transformer attention mechanism",
    filters={
        'publication_year': '2023-2024',
        'type': 'journal-article',
        'open_access.is_oa': 'true'
    },
    max_results=200
)

for paper in papers[:5]:
    print(f"- {paper['title']} ({paper['publication_year']})")
    print(f"  DOI: {paper['doi']}")
    print(f"  Citations: {paper['cited_by_count']}")