RAG Website Crawler
Pre-chunked markdown with token counts for RAG pipelines
What it does
Turn any website into clean, chunked, token-counted markdown — ready to drop into a vector database or LLM context window. Strips boilerplate via Mozilla Readability. Handles SPAs with Playwright. You only pay for pages that actually crawl.
Built in
- ✓ Heading-based chunking with token counts
- ✓ SPA/JavaScript rendering via Playwright
- ✓ Boilerplate stripping (nav, footer, ads)
- ✓ MCP-native for AI agents
- ✓ Zero charge on failure
Guides & tutorials
RAG Crawler Use Cases: Who Needs Website-to-Markdown Conversion and Why
RAG Crawler converts any website into chunked, token-counted markdown. Here are the teams that use it, what they build, and why pre-chunked output matters for LLM pipelines.
Firecrawl vs RAG Crawler: Pricing, Output Quality, and When to Use Each
Firecrawl charges per page on a subscription. RAG Crawler charges per page crawled on pay-per-result. Here is a direct comparison of output, pricing, and failure handling.
The Agentic Data Stack 2025: How to Pick the Right Scrapers for Your AI Workflow
A practical guide to building grounded AI agents with real-time scraped data. Which data sources matter for which agent types
Web Scraping for AI Training Data: Legal, Technical, and Quality Considerations
The complete guide to collecting web-scraped training data for AI models — what is legally permissible, which technical approaches produce quality data
Build a Custom Knowledge Base Chatbot with Claude and the RAG Crawler
Use Apify's RAG Crawler to ingest any website into a vector database, then wire Claude to answer questions against it.
The Best Apify Actors for AI and LLM Projects in 2025
A curated list of Apify actors that ship data in formats LLMs can directly use — ranked by reliability, output quality, and billing fairness.
How to Build a RAG Pipeline Using Web-Scraped Content
A complete guide to turning any website into LLM context — from crawling and chunking to embedding, retrieval, and keeping the index fresh.
Firecrawl Alternative: Web Crawling for RAG Without the $50/Month Tax
Firecrawl is popular but expensive at scale. Here is a direct comparison of every web crawling option for RAG pipelines
Common questions
What is RAG Crawler and what does it do? +
RAG Crawler converts any website into clean, chunked, token-counted markdown ready for vector databases and LLM context windows. It strips navigation and boilerplate using Mozilla Readability, then splits content into heading-based chunks with token counts.
Is RAG Crawler a Firecrawl alternative? +
Yes. RAG Crawler produces the same chunked markdown output as Firecrawl but charges only for pages that successfully crawl. Failed pages cost nothing. For most workloads it is significantly cheaper at scale.
Does it handle JavaScript-heavy websites? +
Yes. RAG Crawler uses Playwright to render JavaScript before extracting content, so single-page apps and dynamically-loaded sites work correctly.
What format does the output use? +
Each page produces a JSON record with the URL, title, markdown content split into chunks, token count per chunk, and total token count. The format is directly compatible with LangChain, LlamaIndex, and OpenAI embedding endpoints.
How does pricing work? +
You pay only for Apify platform compute time. There is no per-page fee from The Mine Works. Pages that fail to crawl do not count toward your compute usage.
Try it on Apify.
Pay only for results delivered. No credit card required to sign up.