Five Scholarly Databases in One Call: OpenAlex, Crossref, arXiv, PubMed and OpenCitations

A proper literature scan is not one search. It is five, each in a different place, each speaking a different language and handing you back a different shape of data.

Try it live: OpenAlex Scraper, 20 Fields, Citations & OA, No API Key. Pay per result delivered. Failed and empty results are never charged.

OpenAlex for published work. Crossref for the canonical record. arXiv for what has not been published yet. PubMed if it is biomedical. OpenCitations when you need to know who actually cited what. Doing all five by hand is why “just do a quick literature review” is never quick.

Here is the same scan in one call, with real output.

One topic, five sources

// tools/call → literature_report
{ "topic": "CRISPR base editing", "from_year": 2022 }

Four sections came back at 25 records each: published works from OpenAlex, canonical metadata from Crossref, arXiv preprints, and PubMed. Crossref returned “CRISPR-Mediated Base Editing Tools: A Genome-Wide Survey” near the top, which is exactly the record you would want.

The part that saves you: knowing which source to trust

The value of having all five side by side is that you learn which one to believe for which question.

On that run, OpenAlex ranked by citation count with fairly loose matching and returned some high-citation papers that were adjacent to the topic rather than squarely on it. Crossref matched the query much more tightly. So the honest reading is: use Crossref when you want precision, and treat the OpenAlex section as a “what is big in this neighbourhood” signal rather than a targeted search. That is not a flaw to hide, it is the kind of thing you can only see when the sources sit next to each other.

The short version: Crossref is the system of record for what a work is, the metadata every publisher registers. OpenAlex is where you go to understand how works relate, who cites whom, what is influential and what is free to read. The other three each answer one narrower question.

Question	Source
What is the canonical DOI, registered title and journal of record?	Crossref
What is most cited, what is open access, and where are the abstracts?	OpenAlex
What has been posted but not yet published?	arXiv
What does the biomedical literature say, with MeSH terms?	PubMed
Who cited this paper, and what does it cite?	OpenCitations

Many pipelines use Crossref to confirm a paper’s identity and OpenAlex to find and enrich everything around it.

The hard half: following a claim forward

Finding papers is the easy half. The hard half is knowing whether a striking result held up, and that lives in the citation record, not in the paper.

The citation-graph tool takes a DOI and returns who cites it and what it cites:

// tools/call → citation_graph
{ "dois": ["10.1038/nature12373"], "direction": "citations" }

That returned 20 citation records, each a paper that built on the original. Walking the forward trail tells you whether a finding is doing real work in the field or is merely being cited in the same shallow way over and over. A paper will never tell you it failed to replicate. The papers that came after it will.

Going deeper on one source

The one-call report is a scan: 25 records a section. When you need a bigger, filtered corpus from one source, run the underlying scraper directly. Each one returns flat JSON, one row per record, and closes the dataset with a single _type: "summary" row carrying the count, so filter that row out before you count anything.

OpenAlex, for scale and abstracts

OpenAlex is usually the one worth going deep on, since it indexes over 250 million works. A literature review pulls the most-cited recent work on a topic, with abstracts:

{
  "searchTerm": "perovskite solar cells",
  "fromYear": 2022,
  "minCitations": 25,
  "includeAbstract": true,
  "maxResults": 300
}

Results come back most-cited first, so the anchor papers sit at the top. minCitations is the simplest noise filter, since it keeps the work the field has actually engaged with rather than every paper that mentions the keyword. OpenAlex stores abstracts as an inverted index, and includeAbstract rebuilds them into readable text.

Drop the citation floor and the same records map a field. Each work carries author_institutions and concepts, so counting them shows who is publishing on a technology and where, which is the raw material for R&D landscaping, partner scouting and talent searches. Counting publication_year gives a sourced answer to whether a field is heating up or cooling down.

import os
from collections import Counter
from apify_client import ApifyClient

apify = ApifyClient(os.environ["APIFY_TOKEN"])

def search_openalex(query: str, from_year: int = 2020, min_citations: int = 0,
                    max_results: int = 100, abstracts: bool = True) -> list[dict]:
    run = apify.actor("themineworks/openalex-scholarly-works").call(run_input={
        "searchTerm": query,
        "fromYear": from_year,
        "minCitations": min_citations,
        "includeAbstract": abstracts,
        "maxResults": max_results,
    })
    items = apify.dataset(run["defaultDatasetId"]).iterate_items()
    return [w for w in items if w.get("_type") != "summary"]


works = search_openalex("perovskite solar cells", from_year=2018,
                        max_results=5000, abstracts=False)

by_institution = Counter(i for w in works for i in w.get("author_institutions") or [])
by_year = Counter(w["publication_year"] for w in works)
print(by_institution.most_common(10))
print(sorted(by_year.items()))

One catch on the trend line: because results are sorted by citations, a capped sample leans toward older papers that have had time to collect them. Set maxResults high enough to cover the whole result set (it goes up to 10,000), or run one query per year. Set openAccessOnly when you want a corpus you can read in full.

PubMed, for biomedical abstracts and MeSH terms

PubMed indexes over 36 million biomedical citations from MEDLINE, life science journals and online books. The NCBI E-utilities API is free and needs no key, but it rate limits, paginates, and makes you run a search call and then a fetch call to get anything useful. The PubMed Scraper runs that sequence (esearch for the PMIDs, then efetch in batches of 50) and parses the XML into flat records.

The inputs are deliberately thin, because PubMed’s own query language already does the filtering:

Input	Default	Notes
query	biomedical research	Passed straight through, so field tags and boolean operators work as they do in the PubMed search box
dateFrom	none	Published from, in PubMed’s own YYYY/MM/DD format
dateTo	none	Published up to, same format
ncbiApiKey	none	Optional free NCBI key. Without one the actor paces at about three requests a second, with one at about nine
maxResults	100	Up to 10,000 articles a run

There is no separate author, journal or publication-type input. Those are field tags inside query:

{
  "query": "semaglutide[Title/Abstract] AND cardiovascular[MeSH Terms]",
  "dateFrom": "2022/01/01",
  "maxResults": 300
}

[Title/Abstract], [MeSH Terms], [Author] and [Journal] all work, as do AND, OR and NOT. dateFrom and dateTo are compiled into a [dp] date-of-publication filter and added to whatever you wrote, so you can combine the two freely.

Eleven fields come back per article:

Field	What it holds
pmid	PubMed identifier, and the stable citation key to store
title	Article title
abstract	Full abstract. Structured abstracts arrive from NCBI as separate labelled sections and are joined into one text field
authors	Author names, forename plus surname
journal	Journal title
issn	Journal ISSN
year	Publication year, as a string
doi	DOI where the record carries one
mesh_terms	MeSH descriptor terms assigned by NLM indexers
url	Link to the PubMed record
scraped_at	ISO capture timestamp

mesh_terms is the field most PubMed exports drop, and it is the one worth having. MeSH is a controlled vocabulary, so counting it tells you what a result set is actually about rather than which words the authors happened to use:

from collections import Counter

def pubmed(query: str, date_from: str | None = None,
           max_results: int = 200) -> list[dict]:
    run_input = {"query": query, "maxResults": max_results}
    if date_from:
        run_input["dateFrom"] = date_from
    run = apify.actor("themineworks/pubmed-ncbi-scraper").call(run_input=run_input)
    items = apify.dataset(run["defaultDatasetId"]).iterate_items()
    return [a for a in items if a.get("_type") != "summary"]


articles = pubmed("GLP-1 receptor agonist cardiovascular outcomes",
                  date_from="2022/01/01", max_results=300)

mesh = Counter(t for a in articles for t in a.get("mesh_terms") or [])
for term, count in mesh.most_common(20):
    print(f"{count:3d}  {term}")

Two things to expect. Very recent articles are often not yet MeSH indexed, so mesh_terms comes back empty on the newest records. And some records carry no abstract or no DOI at all, so those fields come back empty rather than guessed at.

This is the source for a systematic review, where you need a dated, reproducible query you can report in a PRISMA diagram, and for a biomedical RAG corpus, where title plus full abstract plus MeSH terms plus a stable PMID is a complete chunk-and-embed input set. It is also how you track a lab’s or a company’s published output on a molecule or a mechanism, on a schedule.

It costs $0.001 per article delivered on the Apify Free plan and $0.0006 on Gold and above, plus a flat $0.005 when a run starts. A query that matches nothing delivers nothing and charges nothing per article. When NCBI’s own backend fails it returns HTTP 200 with an error buried in the body, which reads like an empty result set and is not one, so the actor reports that as an upstream outage and charges nothing for it. The listing is on Apify.

OpenCitations, for the citation graph

OpenCitations publishes citation links as open data under CC0. The INDEX holds over 1.6 billion of them, drawn from Crossref, PubMed Central and other deposits, and there is no key, no login and no paid tier to clear. The OpenCitations Scraper wraps the v2 API: give it DOIs, get one flat row per citation link.

Input	Default	Notes
dois	none	Array of DOIs. The doi: prefix the v2 API wants is added for you, so either form works
direction	citations	citations for the papers citing yours, references for what it cites, both for each DOI walked in both directions
maxPerDoi	200	Cap per DOI, up to 2,000, so one heavily cited paper cannot consume the whole run

The cap being per DOI rather than per run is the part worth knowing. A landmark paper with 4,000 citing works will not starve the other DOIs in the same list.

Field	What it holds
source_doi	The DOI you asked about
direction	citations or references, which way this row was walked
citing_doi	DOI of the citing paper
cited_doi	DOI of the cited paper
citation_date	OpenCitations’ creation date for the link
journal_sc	Journal self-citation flag, yes or no
author_sc	Author self-citation flag, yes or no
scraped_at	ISO capture timestamp

Those last two flags are the reason to use this source over a plain citation count. A total that quietly includes an author citing their own papers is not an impact measure, and almost nothing else exposes the difference:

import networkx as nx

def citation_links(dois: list[str], direction: str = "citations",
                   max_per_doi: int = 200) -> list[dict]:
    run = apify.actor("themineworks/opencitations-citation-graph").call(run_input={
        "dois": dois,
        "direction": direction,
        "maxPerDoi": max_per_doi,
    })
    items = apify.dataset(run["defaultDatasetId"]).iterate_items()
    return [c for c in items if c.get("_type") != "summary"]


links = citation_links(["10.1038/nature12373"], direction="both", max_per_doi=500)

independent = [c for c in links
               if c.get("author_sc") == "no" and c.get("journal_sc") == "no"]
print(f"{len(links)} links, {len(independent)} after removing self-citations")

G = nx.DiGraph()
for c in independent:
    if c.get("citing_doi") and c.get("cited_doi"):
        G.add_edge(c["citing_doi"], c["cited_doi"])

for doi, count in sorted(G.in_degree(), key=lambda x: x[1], reverse=True)[:10]:
    print(f"{count} citations: {doi}")

scores = nx.pagerank(G)
top_influential = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:10]

In-degree inside that subgraph tells you which papers the neighbourhood keeps returning to. PageRank over the same edges weights a citation by how well cited the citing paper is, which is closer to influence than a raw count.

One gap to plan for: OpenCitations returns DOIs and nothing else, so a graph built from it has no titles in it. The Crossref scraper searches by query, year and work type rather than by a list of DOIs, so to put names on the nodes, resolve each DOI against Crossref’s free REST API and cache what comes back:

import requests

def resolve(doi: str) -> dict | None:
    r = requests.get(f"https://api.crossref.org/works/{doi}",
                     params={"mailto": "you@example.com"}, timeout=20)
    if r.status_code != 200:
        return None
    m = r.json()["message"]
    return {
        "doi": m.get("DOI"),
        "title": (m.get("title") or [None])[0],
        "journal": (m.get("container-title") or [None])[0],
        "year": (m.get("published", {}).get("date-parts") or [[None]])[0][0],
    }

A DOI with no indexed links returns HTTP 404 from OpenCitations, which the actor treats as “no links” rather than as a failure. So a reference list that comes back empty means OpenCitations has no deposit for that paper, not that the paper cites nothing. Coverage is broad but not universal, which makes any count from this source a floor rather than a total.

It costs $0.001 per citation link delivered on the Apify Free plan and $0.0006 on Gold and above, plus the same flat $0.005 run start. DOIs with no indexed links are never charged. The listing is on Apify.

Citation-grounded answers over the abstracts

With abstracts in hand you can build a small retrieval layer that answers research questions and cites the papers it used. Embed the abstracts, store them with their DOIs, and require the model to reference every source.

import anthropic
import chromadb
from openai import OpenAI

openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
claude = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
chroma = chromadb.PersistentClient(path="./research_chroma")
collection = chroma.get_or_create_collection("papers")


def index_papers(papers: list[dict]):
    docs, ids, metas = [], [], []
    for p in papers:
        abstract = p.get("abstract") or ""
        if len(abstract) < 100:
            continue  # nothing useful to embed
        docs.append(f"{p.get('title')}\n\n{abstract}")
        ids.append(p.get("doi") or p["openalex_id"])
        metas.append({
            "title": p.get("title") or "",
            "year": p.get("publication_year") or 0,
            "doi": p.get("doi") or "",
            "citations": p.get("cited_by_count") or 0,
        })
    if not docs:
        return
    embeddings = [e.embedding for e in openai_client.embeddings.create(
        model="text-embedding-3-small", input=docs).data]
    collection.upsert(documents=docs, embeddings=embeddings, ids=ids, metadatas=metas)


def ask(question: str, top_k: int = 6) -> str:
    q_emb = openai_client.embeddings.create(
        model="text-embedding-3-small", input=[question]).data[0].embedding
    hits = collection.query(query_embeddings=[q_emb], n_results=top_k,
                            include=["documents", "metadatas"])
    context = "\n\n---\n\n".join(
        f"[{m['title']} ({m['year']}), DOI {m['doi']}, {m['citations']} citations]\n{d}"
        for d, m in zip(hits["documents"][0], hits["metadatas"][0])
    )
    resp = claude.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1200,
        messages=[{"role": "user", "content": (
            "Answer using only these paper abstracts. Cite every claim as "
            "[Title (Year), DOI]. If the abstracts do not cover it, say so.\n\n"
            f"QUESTION: {question}\n\nABSTRACTS:\n{context}"
        )}],
    )
    return resp.content[0].text


index_papers(search_openalex("retrieval augmented generation",
                             from_year=2022, min_citations=20, max_results=100))
print(ask("What chunking strategies have been shown to improve RAG retrieval quality?"))

The same loop works on PubMed output for a biomedical corpus. Swap openalex_id for pmid as the fallback id, and put mesh_terms in the metadata so you can filter a retrieval by concept rather than by keyword.

Abstracts are enough for literature mapping, gap analysis and “what does the field say about X” questions, because they are written to summarise the contribution. For deep methodological detail you will want the full text of the key papers. Use the abstracts to find which ones are worth fetching, then follow the oa_url where a paper is open access, and respect the publisher’s terms where it is not.

Why one endpoint beats five tabs

Each of these sources has its own scraper. OpenAlex, Crossref and arXiv have their own tutorials too: OpenAlex, Crossref DOI metadata and the arXiv preprint scraper. PubMed and OpenCitations are covered above, with their own pages at PubMed Scraper and OpenCitations Scraper. Running them one at a time and reconciling five formats is the slow way.

The Academic Research MCP server puts all five behind a single endpoint your AI agent can call, so Claude pulls the whole scan in one step and reasons over it. It is the cheapest server we run, because every source is a free public API with no proxy fight and no login. You pay per result, and a topic that returns nothing adds no result charges.

Honest limits

These are metadata and abstract services, not full-text retrieval. Citation counts differ between sources because their corpora differ, and OpenCitations in particular is a floor rather than a total, since a citation only appears once someone deposits it. PubMed is biomedical only, and the report marks that section not-found for out-of-domain topics rather than padding it, while its newest records are often not yet MeSH indexed. And as above, OpenAlex relevance is loose on narrow queries, which is exactly why having Crossref in the same call matters.

Ready to try it? The Academic Research MCP runs on Apify with your own account. Setup for Claude, Cursor and Windsurf is in the listing.