arxiv-latex-source

1

brycewang-stanford/Auto-Empirical-Research-SkillsResearchAudit passedSnapshot f5c1d0167ef1

Installs: 0

Summary

Download and parse LaTeX source files from arXiv preprints

SKILL.md

arXiv LaTeX Source Access Guide

Overview

arXiv stores the original LaTeX source files for the vast majority of its 2.4 million+ preprints. Accessing LaTeX source provides major advantages over PDF parsing: exact mathematical notation as written by the author, structured sections and labels, machine-readable bibliography entries, and intact figure captions, table data, and cross-references.

For formula extraction, citation graph construction, section-level text analysis, or training data curation for scientific language models, LaTeX source is the gold standard. PDF parsing introduces OCR errors in equations, loses structural hierarchy, and mangles complex tables.

The e-print endpoint serves source bundles as gzip-compressed tarballs (.tar.gz) containing .tex files, figures, .bib/.bbl bibliography files, style files, and supplementary materials. No authentication is required.

Authentication

No authentication or API key is required. The e-print endpoint is publicly accessible. However, arXiv asks that automated tools set a descriptive User-Agent header and comply with rate limits.

Core Endpoints

Download LaTeX Source

URL: GET https://arxiv.org/e-print/{arxiv_id}
Response: application/gzip — a .tar.gz archive containing the source files
Parameters:

Param Type Required Description
arxiv_id string Yes arXiv identifier, e.g. 2301.00001 or 2301.00001v2 for a specific version

Example:

# Download source archive (response: 200, application/gzip, ~1.3 MB)
curl -sL -o source.tar.gz "https://arxiv.org/e-print/2301.00001"

# List archive contents
tar tz -f source.tar.gz | head -10
# ACM-Reference-Format.bbx
# ACM-Reference-Format.bst
# Image_1.jpg
# README.txt
# acmart.cls

curl -sL "https://arxiv.org/e-print/{arxiv_id}" -o source.gz
file source.gz  # "gzip compressed data, was 'XXXX.tar', ..."

import tarfile, re

def find_main_tex(tar_path):
    with tarfile.open(tar_path, 'r:gz') as tar:
        tex_files = [m for m in tar.getmembers() if m.name.endswith('.tex')]
        for member in tex_files:
            content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
            if r'\documentclass' in content and r'\begin{document}' in content:
                return member.name, content
    return None, None

import re

def extract_sections(tex_content):
    pattern = r'\\(section|subsection|subsubsection)\{([^}]+)\}'
    sections = re.findall(pattern, tex_content)
    return [(level, title) for level, title in sections]
    # [('section', 'Introduction'), ('section', 'Related Work'), ...]

def extract_equations(tex_content):
    patterns = [
        r'\\\[(.+?)\\\]',
        r'\\begin\{equation\}(.+?)\\end\{equation\}',
        r'\\begin\{align\*?\}(.+?)\\end\{align\*?\}',
    ]
    equations = []
    for pat in patterns:
        equations.extend(re.findall(pat, tex_content, re.DOTALL))
    return equations

def extract_bibliography(tar_path):
    refs = []
    with tarfile.open(tar_path, 'r:gz') as tar:
        for member in tar.getmembers():
            if member.name.endswith('.bib'):
                content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
                refs.extend(re.findall(r'@\w+\{([^,]+),(.+?)\n\}', content, re.DOTALL))
            elif member.name.endswith('.bbl'):
                content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
                refs.extend(re.findall(r'\\bibitem.*?\{(.+?)\}', content))
    return refs

import requests, tarfile, io, re, time, gzip

def download_arxiv_source(arxiv_id, delay=1.0):
    """Download and extract all .tex files from an arXiv paper's source."""
    url = f"https://arxiv.org/e-print/{arxiv_id}"
    headers = {"User-Agent": "ResearchTool/1.0 (mailto:[email protected])"}
    resp = requests.get(url, headers=headers)
    resp.raise_for_status()
    time.sleep(delay)

    buf = io.BytesIO(resp.content)
    try:
        with tarfile.open(fileobj=buf, mode='r:gz') as tar:
            return {m.name: tar.extractfile(m).read().decode('utf-8', errors='ignore')
                    for m in tar.getmembers() if m.name.endswith('.tex') and m.isfile()}
    except tarfile.ReadError:
        buf.seek(0)
        return {"main.tex": gzip.decompress(buf.read()).decode('utf-8', errors='ignore')}

# Usage
sources = download_arxiv_source("2301.00001")
for fname, content in sources.items():
    if r'\documentclass' in content:
        sections = re.findall(r'\\section\{([^}]+)\}', content)
        equations = re.findall(r'\\begin\{equation\}(.+?)\\end\{equation\}', content, re.DOTALL)
        print(f"{fname}: {len(sections)} sections, {len(equations)} equations")