RAG Website Crawler
Pre-chunked markdown with token counts for RAG pipelines
What it does
Turn any website into clean, chunked, token-counted markdown, ready to drop into a vector database or LLM context window. Strips boilerplate via Mozilla Readability. Handles SPAs with Playwright. You only pay for pages that actually crawl.
Built in
- ✓ Heading-based chunking with token counts
- ✓ SPA/JavaScript rendering via Playwright
- ✓ Boilerplate stripping (nav, footer, ads)
- ✓ MCP-native for AI agents
- ✓ Failed results are never charged
Guides & tutorials
Firecrawl vs RAG Crawler: Pricing, Output Quality, and When to Use Each
Firecrawl charges per page on a subscription. RAG Crawler charges per page crawled on pay-per-result. Here is a direct comparison of output, pricing, and failure handling.
The Agentic Data Stack 2025: How to Pick the Right Scrapers for Your AI Workflow
A practical guide to building grounded AI agents on live scraped data, and which data sources matter for which kind of agent.
Web Scraping for AI Training Data: Legal, Technical, and Quality Considerations
A guide to collecting web-scraped AI training data: what is legally permissible and which technical approaches produce quality data.
The Best Apify Actors for AI and LLM Projects in 2025
A curated list of Apify actors that ship data in formats LLMs can directly use, ranked by reliability, output quality, and billing fairness.
How to Build a RAG Pipeline Using Web-Scraped Content
A complete guide to turning any website into LLM context, from crawling and chunking to embedding, retrieval, and keeping the index fresh.
Related scrapers
YouTube Transcript Scraper
Timestamped captions to JSON. No API key, no login
GitHub Repo Intelligence
Deep metadata for any public GitHub repo: stars, README, releases
GitHub Skill & Memory Discovery
Find every Claude/AI agent skill and memory repo on GitHub, full content
GitHub Trending Scraper
github.com/trending by language and time window. The only reliable API
Common questions
What is RAG Crawler and what does it do? +
RAG Crawler converts any website into clean, chunked, token-counted markdown ready for vector databases and LLM context windows. It strips navigation and boilerplate using Mozilla Readability, then splits content into heading-based chunks with token counts.
Is RAG Crawler a Firecrawl alternative? +
Yes. RAG Crawler produces the same chunked markdown output as Firecrawl but charges only for pages that successfully crawl. Failed pages are never charged. For most workloads it is significantly cheaper at scale.
Does it handle JavaScript-heavy websites? +
Yes. RAG Crawler uses Playwright to render JavaScript before extracting content, so single-page apps and dynamically-loaded sites work correctly.
What format does the output use? +
Each page produces a JSON record with the URL, title, markdown content split into chunks, token count per chunk, and total token count. The format is directly compatible with LangChain, LlamaIndex, and OpenAI embedding endpoints.
How does pricing work? +
Pay per result: from $1.00 per 1,000 pages on Apify's higher plans, $2.00 on the free plan, plus a $0.005 start fee per run. Pages that fail to crawl are never charged.
Try it on Apify.
Pay only for results delivered. No credit card required to sign up.