PubMed Scraper: Search 36M Biomedical Articles, Abstracts, and MeSH Terms via API

PubMed indexes over 36 million biomedical citations from MEDLINE, life science journals, and online books. The NCBI E-utilities API provides free access but requires managing rate limits, pagination, and multi-step fetch flows. Our PubMed Scraper handles all of that and returns flat JSON records ready for analysis.

What you get per article

• pmid — PubMed identifier
• title — article title
• abstract — full abstract text
• authors — list of author names
• journal — journal name and abbreviation
• pub_date — publication date
• doi — DOI if available
• mesh_terms — Medical Subject Headings assigned by NLM indexers
• publication_types — article types (Clinical Trial, Review, Meta-Analysis, etc.)
• pmc_id — PubMed Central ID if full text is open access
• scraped_at — ISO timestamp

Python example

import requests, time

run = requests.post(
    "https://api.apify.com/v2/acts/themineworks~pubmed-ncbi-scraper/runs",
    headers={"Authorization": f"Bearer {APIFY_TOKEN}"},
    json={
        "query": "GLP-1 receptor agonist cardiovascular outcomes",
        "dateFrom": "2022-01-01",
        "publicationType": "Clinical Trial",
        "maxResults": 100
    }
)

run_id = run.json()["data"]["id"]

while True:
    r = requests.get(
        f"https://api.apify.com/v2/actor-runs/{run_id}",
        headers={"Authorization": f"Bearer {APIFY_TOKEN}"}
    ).json()
    if r["data"]["status"] in ("SUCCEEDED", "FAILED"):
        break
    time.sleep(5)

articles = requests.get(
    f"https://api.apify.com/v2/actor-runs/{run_id}/dataset/items",
    headers={"Authorization": f"Bearer {APIFY_TOKEN}"}
).json()

for a in articles:
    print(a["pmid"], a["pub_date"])
    print(f"  {a['title']}")
    print(f"  {a['journal']} | DOI: {a.get('doi','—')}")

MeSH term analysis

from collections import Counter
import pandas as pd

all_mesh = []
for article in articles:
    all_mesh.extend(article.get("mesh_terms", []))

mesh_freq = Counter(all_mesh).most_common(20)
print("Top MeSH terms in this result set:")
for term, count in mesh_freq:
    print(f"  {count:3d}  {term}")

Build a RAG corpus from PubMed

# Collect abstracts for a topic and load into a vector store
corpus = []
for a in articles:
    if a.get("abstract"):
        corpus.append({
            "id": a["pmid"],
            "text": f"{a['title']}\n\n{a['abstract']}",
            "metadata": {
                "pmid": a["pmid"],
                "journal": a["journal"],
                "pub_date": a["pub_date"],
                "doi": a.get("doi"),
            }
        })

print(f"{len(corpus)} documents ready for embedding")

Parameters

Parameter	Default	Notes
query	required	PubMed search query (supports field tags like [tiab], [au], [mh])
author	—	Author name filter
meshTerm	—	MeSH heading filter
publicationType	—	Filter to Clinical Trial, Review, Meta-Analysis, etc.
dateFrom	—	Published after date (YYYY-MM-DD)
maxResults	100	Maximum articles to return

PubMed query syntax

PubMed supports field-tagged queries:

cancer[tiab] AND immunotherapy[tiab]        # Title/abstract
Smith J[au]                                  # Author
"diabetes mellitus"[mh]                      # MeSH heading
2023:2025[dp]                               # Date range

Leave the query blank and specify only meshTerm to pull all articles indexed under a heading.

Use cases

Systematic literature review. Run a PICO-structured query to pull all relevant trials for a Cochrane-style review. Export to CSV for PRISMA screening.

Biomedical RAG pipeline. Feed PubMed abstracts into a vector database to power a medical literature Q&A tool or clinical decision support prototype.

Competitive research intelligence. Track publications from specific research groups, institutions, or pharmaceutical companies to monitor their scientific output and pipeline direction.

Grant landscape analysis. Pull recent publications in a research area to identify active investigators, landmark papers, and open questions for a grant application background section.

Drug discovery literature mining. Search for abstracts mentioning a target protein or pathway to aggregate evidence before committing to a discovery program.

Pricing

$0.002 per article. Empty queries are not charged.

Available on Apify.