Pay-Per-Result vs Subscription Scraping: Why Billing Models Matter More Than You Think

The billing model of a scraping tool determines your cost structure in ways that compound at scale. A 20% failure rate, typical for sites with bot protection, changes the economics of per-run billing dramatically. This post makes the math explicit.

TL;DR: Pay-per-result (PPE) billing charges for successfully returned data: failed and empty results are never charged, and at most a small start fee applies per run. At a 15% failure rate the difference is modest. At 30% failure (common with anti-bot sites), PPE is 20-40% cheaper than per-run billing. More importantly, PPE creates a financial incentive for the scraper developer to maximize reliability. They only earn when you get data.

The Three Billing Models

Per-month subscription: You pay $X/month regardless of how much data you actually get. Examples: Firecrawl ($19-199/month), ScraperAPI ($49-399/month).

Per-run: You pay for compute time or API calls regardless of whether data is returned. This is how most Apify actors are billed by default.

Per-result (PPE): You pay per result successfully returned, plus at most a small start fee per run. Failed and empty results are never charged.

The Failure Rate Problem

Web scraping is not deterministic. Sites go down. Bot protection blocks individual requests. Pages take too long to load. Pagination fails halfway through.

Real-world failure rates for common targets:

• Static, no-bot-protection sites: 1-3% failure
• Sites with Cloudflare or basic bot detection: 10-25% failure
• Sites with Akamai Bot Manager (Naukri, major banks): 15-30% failure without proper stealth
• Sites actively blocking scrapers (LinkedIn): 30-60% failure

Under per-run billing, you pay full price for failed runs. Under PPE billing, a failed run costs you at most its small start fee.

The Math at Scale

Scenario: You run a job data pipeline that scrapes 10,000 jobs per month from Naukri (15% failure rate in a well-implemented scraper).

Per-run billing at $0.005/compute-unit, 0.5 CU per run:

• 10,000 successful jobs
• ~1,765 failed runs at the same cost
• Total cost: 11,765 × $0.0025 = $29.41

PPE billing at $0.003/result:

• 10,000 successful jobs × $0.003 = $30.00

At low failure rates, the difference is small. Now crank the failure rate to 30% (a realistic number for anti-bot bypass attempts without residential proxies):

Per-run billing:

• 10,000 successful + 4,286 failed = 14,286 runs × $0.0025 = $35.71

PPE billing:

• 10,000 × $0.003 = $30.00 (unchanged, failures are free)

The cost difference grows with the failure rate. For targets with serious bot protection, PPE billing can be 20-40% cheaper.

What These Models Are Called Elsewhere

The same handful of models go by different names depending on who is selling them, which makes like-for-like comparison harder than it should be.

Pay as you go usually means pay-per-result when a scraper vendor uses it, but not always, and the difference is the whole ballgame. Some vendors use it to mean metered compute: billed per minute, per request or per gigabyte whether or not a row came back. Read the unit before assuming. If the unit is a row, a record or a result, failures are free. If the unit is time, memory or a request, you are paying for failures under a friendlier name.

Flat rate is subscription billing with the cap stated out loud: a fixed monthly fee for a set volume. Two questions decide whether it is cheap. What happens to unused volume at the end of the month, and what does the overage cost when a job runs long.

Credits are the most opaque of the three, because a credit is a synthetic unit the vendor defines and can redefine. A credit that buys 10 results on a static site and 1 on a protected one is a per-run model wearing a costume.

The test that cuts through all of it: ask what a failed request costs. Every model answers that question differently, and the answer is the cost structure.

Self-Hosted vs Managed Scraping

Billing model is one axis. Where the scraper runs is the other, and people usually compare them badly.

Self-hosting looks cheaper because the visible cost is a server. The costs that are not on the invoice are proxy rotation, the engineering time to keep selectors working as sites change, and the on-call burden when a target adds bot protection on a Tuesday. For a handful of stable, unprotected sites, self-hosting genuinely is cheaper and stays that way.

Managed makes sense when the target fights back. The maintenance is the product: a scraper against a Cloudflare or Akamai protected site is not written once, it is maintained continuously, and that cost lands on whoever owns the code.

For high-frequency pipelines the deciding factor is usually neither price nor control but blast radius. If a target changes its markup at 2am, self-hosted means your pipeline is down until someone wakes up.

Beyond Cost: The Incentive Structure Difference

PPE billing creates a different incentive structure for the actor developer.

Under per-run billing: The developer earns the same whether the run succeeds or fails. There is no financial incentive to improve reliability.

Under PPE billing: The developer earns no result revenue on failed runs. Every percentage point of reliability improvement directly increases developer revenue.

This is why the actors we build use PPE billing exclusively. We are only paid when you get data.

Subscription Billing: When It Makes Sense

Subscription billing is not always worse. It makes sense when:

1. High volume, predictable usage: If you are crawling 50,000 pages per month every month, a subscription at $199/month is often cheaper than per-page pricing at $0.005/page ($250).

2. Included infrastructure: ScraperAPI’s subscription includes proxy infrastructure, JavaScript rendering, and automatic retry logic. The “overhead” in subscription pricing often includes real value.

3. SLA requirements: Enterprise subscription plans include uptime guarantees and support SLAs that per-result billing typically does not.

The Hidden Cost of Subscription Billing

The scenario that makes subscription billing expensive: variable or unpredictable usage.

If you are building a new product and collecting data weekly while the product grows from 10 to 100 to 1,000 customers, a fixed subscription means you are overpaying during low-usage periods.

PPE billing scales with your actual usage. You never pay for capacity you are not using.

Our Billing Model

Every actor we publish uses PPE billing on Apify’s platform. The event name is [actor-name]-result-scraped, charged at the rate defined in the actor’s monetization configuration.

If you run our Reddit Scraper and it returns 847 posts out of 1,000 requested, you pay for 847, plus the run’s small start fee. The 153 failed attempts cost nothing.

This is the only billing model we are willing to build on. If we cannot reliably deliver results, we should not be paid.

Frequently Asked Questions

What is PPE billing in Apify?

PPE (pay-per-event) is Apify’s billing model where actors charge per result successfully returned rather than per compute unit consumed. The actor developer defines an event name and price per event in their actor’s monetization settings. If a run returns zero results, the user pays nothing. All The Mine Works actors use PPE billing exclusively.

How does scraping failure rate affect total cost under different billing models?

Under per-run billing, you pay for both successful and failed runs. At a 15% failure rate on 10,000 target results, you pay for ~11,765 runs. At 30% failure, you pay for 14,286 runs, 43% more than the equivalent pay-per-result cost. PPE billing keeps your cost nearly constant regardless of failure rate, because failed results are never charged and only a small per-run start fee is fixed.

When is subscription billing better than pay-per-result?

Subscription billing wins for predictable, high-volume usage: if you crawl 50,000+ pages every month consistently, a fixed subscription is often cheaper than per-page pricing. It also makes sense when the subscription bundles real additional value: proxy infrastructure, SLA guarantees, and dedicated support.

Why does PPE billing create better reliability incentives for scraper developers?

Under per-run billing, a developer earns the same revenue whether the run succeeds or fails. There is no financial incentive to maximize reliability. Under PPE billing, a failed run earns the developer no result revenue. This directly aligns developer earnings with delivering reliable data, rather than just processing requests.

What is a realistic web scraping failure rate to expect?

Failure rates vary significantly by target: 1-3% for static sites with no bot protection, 10-25% for Cloudflare-protected sites, 15-30% for Akamai-protected enterprise sites like Naukri or major banks without session warming, and 30-60% for aggressively protected targets like LinkedIn.

What is the difference between pay as you go and pay-per-result?

Often nothing, but check the billing unit. Both names get used for result-based billing, where failed and empty results are never charged. Some vendors use pay as you go for metered compute instead, billing per minute or per request regardless of output. The unit on the invoice tells you which one you bought.

Is a flat rate scraper API cheaper than per-result billing?

It depends entirely on how much of the allowance you use. Flat rate is a bet that your volume is predictable and high. If usage is spiky, or if a chunk of your target list is protected and fails often, the unused allowance and the overage rate are where the money goes.

Is self-hosted scraping cheaper than a managed scraper API?

For a small number of stable, unprotected sites, usually yes. The comparison changes once targets have bot protection, because the real cost is ongoing maintenance rather than servers, and that cost does not appear on any invoice.