arXiv Scraper: Search AI, Physics, and Biology Preprints with PDF Links via API

arXiv hosts over 2.3 million preprints across physics, mathematics, computer science, quantitative biology, economics, and statistics. In AI and ML, it is effectively the primary publication venue — papers appear on arXiv weeks or months before journal publication. Our arXiv Scraper queries the full index and returns structured records with PDF links.

What you get per paper

• arxiv_id — arXiv identifier (e.g. 2301.07543)
• title — paper title
• abstract — full abstract text
• authors — list of author names
• categories — arXiv subject categories (e.g. cs.LG, cs.CL)
• primary_category — primary category
• submitted — submission date
• updated — last update date
• pdf_url — direct link to the PDF
• abs_url — arXiv abstract page URL
• doi — DOI if published in a journal
• scraped_at — ISO timestamp

Python example

import requests, time

run = requests.post(
    "https://api.apify.com/v2/acts/themineworks~arxiv-preprint-search/runs",
    headers={"Authorization": f"Bearer {APIFY_TOKEN}"},
    json={
        "query": "mixture of experts language model scaling",
        "category": "cs.LG",
        "dateFrom": "2024-01-01",
        "maxResults": 100
    }
)

run_id = run.json()["data"]["id"]

while True:
    r = requests.get(
        f"https://api.apify.com/v2/actor-runs/{run_id}",
        headers={"Authorization": f"Bearer {APIFY_TOKEN}"}
    ).json()
    if r["data"]["status"] in ("SUCCEEDED", "FAILED"):
        break
    time.sleep(5)

papers = requests.get(
    f"https://api.apify.com/v2/actor-runs/{run_id}/dataset/items",
    headers={"Authorization": f"Bearer {APIFY_TOKEN}"}
).json()

for p in papers:
    print(p["arxiv_id"], p["submitted"])
    print(f"  {p['title']}")
    print(f"  {p['pdf_url']}")

Monitor AI research weekly

from datetime import date, timedelta

last_week = (date.today() - timedelta(days=7)).isoformat()

run = requests.post(
    "https://api.apify.com/v2/acts/themineworks~arxiv-preprint-search/runs",
    headers={"Authorization": f"Bearer {APIFY_TOKEN}"},
    json={
        "category": "cs.AI",
        "dateFrom": last_week,
        "maxResults": 200
    }
)
# schedule this weekly to track the frontier

Build a research monitoring feed

import pandas as pd

df = pd.DataFrame(papers)
df["submitted"] = pd.to_datetime(df["submitted"])

# Papers per week
weekly = df.groupby(df["submitted"].dt.to_period("W")).size()
print(weekly)

# Most prolific authors in this result set
from collections import Counter
all_authors = [a for p in papers for a in p.get("authors", [])]
print(Counter(all_authors).most_common(15))

Parameters

Parameter	Default	Notes
query	—	Keyword search across title and abstract
author	—	Author name filter
category	—	arXiv category code (e.g. cs.LG, cs.CL, quant-ph, q-bio.GN)
dateFrom	—	Submitted after date (YYYY-MM-DD)
sortBy	"submittedDate"	Sort by "submittedDate" or "relevance"
maxResults	100	Maximum papers to return

arXiv category codes

Code	Field
cs.LG	Machine Learning
cs.CL	Computation and Language (NLP)
cs.CV	Computer Vision
cs.AI	Artificial Intelligence
stat.ML	Statistics: Machine Learning
q-bio.GN	Genomics
physics.bio-ph	Biological Physics

Use cases

AI research monitoring. Track the frontier of a specific subfield — LLM reasoning, diffusion models, multimodal learning — as papers appear, before they reach conference proceedings.

Literature-grounded RAG. Pull abstracts for a research domain and build a retrieval-augmented QA system for R&D teams who need to query the current research landscape.

Competitive intelligence for AI companies. Monitor preprints from competitor research labs. Publication velocity and topic clusters signal where teams are investing.

Academic citation seeding. Find the latest foundational papers to cite in a grant or paper submission before a database like PubMed has indexed them.

Investment research. Track which institutions are publishing most actively in an emerging area (quantum computing, protein folding, materials science) as a signal for talent concentration and licensing potential.

Pricing

$0.002 per paper. Empty queries are not charged.

Available on Apify.