Pay-Per-Result vs Subscription Scraping: Why Billing Models Matter More Than You Think

The billing model of a scraping tool determines your cost structure in ways that compound at scale. A 20% failure rate — typical for sites with bot protection — changes the economics of per-run billing dramatically. This post makes the math explicit.

TL;DR: Pay-per-result (PPE) billing charges only for successfully returned data — failed runs cost nothing. At a 15% failure rate the difference is modest. At 30% failure (common with anti-bot sites), PPE is 20–40% cheaper than per-run billing. More importantly, PPE creates a financial incentive for the scraper developer to maximize reliability — they only earn when you get data.

The Three Billing Models

Per-month subscription: You pay $X/month regardless of how much data you actually get. Examples: Firecrawl ($19-199/month), ScraperAPI ($49-399/month).

Per-run: You pay for compute time or API calls regardless of whether data is returned. This is how most Apify actors are billed by default.

Per-result (PPE): You pay only when data is successfully returned. Failed runs cost nothing.

The Failure Rate Problem

Web scraping is not deterministic. Sites go down. Bot protection blocks individual requests. Pages take too long to load. Pagination fails halfway through.

Real-world failure rates for common targets:

• Static, no-bot-protection sites: 1-3% failure
• Sites with Cloudflare or basic bot detection: 10-25% failure
• Sites with Akamai Bot Manager (Naukri, major banks): 15-30% failure without proper stealth
• Sites actively blocking scrapers (LinkedIn): 30-60% failure

Under per-run billing, you pay for failed runs. Under PPE billing, failures are the platform’s problem, not yours.

The Math at Scale

Scenario: You run a job data pipeline that scrapes 10,000 jobs per month from Naukri (15% failure rate in a well-implemented scraper).

Per-run billing at $0.005/compute-unit, 0.5 CU per run:

• 10,000 successful jobs
• ~1,765 failed runs at the same cost
• Total cost: 11,765 × $0.0025 = $29.41

PPE billing at $0.003/result:

• 10,000 successful jobs × $0.003 = $30.00

At low failure rates, the difference is small. Now crank the failure rate to 30% (a realistic number for anti-bot bypass attempts without residential proxies):

Per-run billing:

• 10,000 successful + 4,286 failed = 14,286 runs × $0.0025 = $35.71

PPE billing:

• 10,000 × $0.003 = $30.00 (unchanged — failures are free)

The cost difference grows with the failure rate. For targets with serious bot protection, PPE billing can be 20-40% cheaper.

Beyond Cost: The Incentive Structure Difference

PPE billing creates a different incentive structure for the actor developer.

Under per-run billing: The developer earns the same whether the run succeeds or fails. There is no financial incentive to improve reliability.

Under PPE billing: The developer earns zero on failed runs. Every percentage point of reliability improvement directly increases developer revenue.

This is why the actors we build use PPE billing exclusively. We are only paid when you get data.

Subscription Billing: When It Makes Sense

Subscription billing is not always worse. It makes sense when:

1. High volume, predictable usage: If you are crawling 50,000 pages per month every month, a subscription at $199/month is often cheaper than per-page pricing at $0.005/page ($250).

2. Included infrastructure: ScraperAPI’s subscription includes proxy infrastructure, JavaScript rendering, and automatic retry logic. The “overhead” in subscription pricing often includes real value.

3. SLA requirements: Enterprise subscription plans include uptime guarantees and support SLAs that per-result billing typically does not.

The Hidden Cost of Subscription Billing

The scenario that makes subscription billing expensive: variable or unpredictable usage.

If you are building a new product and collecting data weekly while the product grows from 10 to 100 to 1,000 customers, a fixed subscription means you are overpaying during low-usage periods.

PPE billing scales with your actual usage. You never pay for capacity you are not using.

Our Billing Model

Every actor we publish uses PPE billing on Apify’s platform. The event name is [actor-name]-result-scraped, charged at the rate defined in the actor’s monetization configuration.

If you run our Reddit Scraper and it returns 847 posts out of 1,000 requested, you pay for 847. The 153 failed attempts cost nothing.

This is the only billing model we are willing to build on. If we cannot reliably deliver results, we should not be paid.

Frequently Asked Questions

What is PPE billing in Apify?

PPE (pay-per-event) is Apify’s billing model where actors charge per result successfully returned rather than per compute unit consumed. The actor developer defines an event name and price per event in their actor’s monetization settings. If a run returns zero results, the user pays nothing. All The Mine Works actors use PPE billing exclusively.

How does scraping failure rate affect total cost under different billing models?

Under per-run billing, you pay for both successful and failed runs. At a 15% failure rate on 10,000 target results, you pay for ~11,765 runs. At 30% failure, you pay for 14,286 runs — 43% more than the equivalent pay-per-result cost. PPE billing keeps your cost constant regardless of failure rate, because failures cost nothing.

When is subscription billing better than pay-per-result?

Subscription billing wins for predictable, high-volume usage — if you crawl 50,000+ pages every month consistently, a fixed subscription is often cheaper than per-page pricing. It also makes sense when the subscription bundles real additional value: proxy infrastructure, SLA guarantees, and dedicated support.

Why does PPE billing create better reliability incentives for scraper developers?

Under per-run billing, a developer earns the same revenue whether the run succeeds or fails — there is no financial incentive to maximize reliability. Under PPE billing, every failed run earns zero revenue for the developer. This directly aligns developer earnings with delivering reliable data, rather than just processing requests.

What is a realistic web scraping failure rate to expect?

Failure rates vary significantly by target: 1–3% for static sites with no bot protection, 10–25% for Cloudflare-protected sites, 15–30% for Akamai-protected enterprise sites like Naukri or major banks without session warming, and 30–60% for aggressively protected targets like LinkedIn.