Documentation

One endpoint. Send a URL and get back rendered HTML, clean text, markdown, a screenshot, or JSON shaped by your own selectors. A browser and a rotating proxy pool sit behind it. Already integrated against ScrapingBee? Parameter names, defaults and credit costs match theirs, so it's a base URL and a key. See Migrating from ScrapingBee.

Quick start

Every request needs a key. A free account starts with 1,000 credits, gets 250 more every month (banking up to 1,000), and needs only an email address.

curl "https://api.webfetch.io/v1/?url=https%3A%2F%2Fexample.com&render_js=false" \
  -H "Authorization: Bearer wfk_your_key"

Asking for structured data instead of HTML:

curl "https://api.webfetch.io/v1/" \
  -H "Authorization: Bearer wfk_your_key" \
  --data-urlencode "url=https://quotes.toscrape.com/js/" \
  --data-urlencode 'extract_rules={"quotes":{"selector":".quote","type":"list","output":{"text":".text","author":".author"}}}' \
  -G

Both hosts serve the same API:

https://api.webfetch.io/v1/
https://webfetch.io/api/v1/

Authentication

Send your key as a bearer token:

Authorization: Bearer wfk_your_key

The api_key query parameter also works, because ScrapingBee accepts it and migrating code often uses it. It is the weaker option: query strings end up in logs and referrers, so prefer the header.

There is no keyless tier. Rate limiting alone does not hold against a large address pool, and an anonymous endpoint leaves no way to act against whoever abuses it. A request without a key returns 401 with code key_required.

Parameters

Sent as a query string on GET, or as a JSON or form body on POST. Query parameters still apply on a POST.

ParameterTypeDefaultDescription
api_key string required Your key. Prefer the Authorization: Bearer header; this query parameter is accepted for compatibility.
url string required The URL to fetch, URL-encoded.
render_js boolean true Render in a browser. Set false for a plain HTTP fetch at 1 credit instead of 5.
wait integer ms 0 Extra settle time after load. Maximum 35000.
wait_for string "" CSS or XPath selector to wait for before capturing.
wait_browser string domcontentloaded One of domcontentloaded, load, networkidle0, networkidle2.
js_scenario stringified JSON {} Click, fill, scroll and wait before capture. See below.
window_width integer 1920 Viewport width, 200 to 3840.
window_height integer 1080 Viewport height, 200 to 2160.
premium_proxy boolean false Route through static ISP proxies: addresses registered to internet providers rather than hosting companies, which sites trust more than datacenter IPs. 10 credits, or 25 with rendering. With no ISP proxy available the call falls back to the residential pool (same price), then a datacenter IP at the base rate, and says so in Wf-Proxy-Downgraded and Wf-Proxy-Pool. Note for ScrapingBee users: their premium_proxy is residential.
stealth_proxy boolean false For targets that refuse ordinary traffic. Renders in our stealth browser, built to get past bot checks: it runs the page's scripts with real mouse and scroll input and loads every resource, takes its time zone and language from the exit, and costs 75 credits. A request it cannot take (with cookies, forwarded headers, js_scenario, wait_for, block_resources=true sent explicitly, block_ads, device=mobile, another window size, return_page_source or screenshot_selector) renders in our hardened standard browser instead, billed for the exit used. It goes out on a datacenter exit first; if the site refuses it (an error, a block status or a challenge page), it is tried once more on an ISP exit. Implies render_js. Billed for the exit that answered when the stealth browser could not take it: the base rate (5) on datacenter, the premium rate (25) on ISP or residential, 75 on a mobile exit. Wf-Proxy-Pool names the pool used and Wf-Proxy-Downgraded: true marks a call billed below 75, so you are never charged for more than you got. Capped at 5 MB per scrape on the mobile and residential pools.
country_code string "" Two-letter ISO 3166-1 code for the exit country. A plain request exits from the datacenter pool's countries at no extra cost; with premium_proxy or stealth_proxy it uses that pool's countries (see the network page). A country the route cannot exit from is refused with 400 country_unavailable rather than served from another country.
own_proxy string "" Use your own proxy: [http://]<user>:<pass>@<host>:<port>. Port defaults to 1080. Only HTTP proxies are supported right now: https and socks proxies are refused with 400 own_proxy_unsupported.
session_id integer "" Route requests sharing this id through the same exit IP.
block_resources boolean true Skip images, media, fonts and stylesheets, and strip third-party telemetry: ad exchanges, analytics, tag managers, error reporting, consent banners and social embed widgets. Measured across major news and retail sites this removes about 37% of the bytes while preserving 100% of the text, headings and links, because none of it carries page content. The main document is never blocked whatever its host, and the apex domains customers scrape are never on the list. Set false to load the page exactly as a browser would.
block_ads boolean false Strip the same telemetry and ad hosts even when block_resources is off. Redundant at the default, kept because it is a ScrapingBee parameter.
extract_rules stringified JSON "" CSS or XPath selectors to structured JSON. See below.
return_page_source boolean false Return the HTML as delivered, before JavaScript ran.
return_page_text boolean false Return readable text with menus, headers and footers removed.
return_page_markdown boolean false Return the content as markdown, with the navigation, header, footer and banners removed and links made absolute. See URL to Markdown.
json_response boolean false Wrap everything in a JSON envelope instead of returning the body raw.
add_html boolean false With json_response=true and a transformation (extract_rules, return_page_text or return_page_markdown), also include the untransformed HTML in body. A request with no transformation already returns it either way.
screenshot boolean false Also capture a screenshot. Returned as a link, at no extra credit cost. Not available together with js_scenario or stealth_proxy right now (400 screenshot_unsupported).
screenshot_full_page boolean false Capture the whole scrollable page.
screenshot_selector string "" Report the bounding box of this selector with the screenshot.
cookies string "" Cookies to send: name=value,domain=example.com;other=value2.
forward_headers boolean false Forward headers you prefix with Wf- (or Spb-).
forward_headers_pure boolean false Forward only your headers, adding none of ours.
device string desktop desktop or mobile.
timeout integer ms 140000 Budget for the fetch, as at ScrapingBee. Whatever it says, a call is answered within 90 seconds in total, mode=auto routes included, because the network edge in front of the API closes longer connections. A request that runs out of time returns a timeout error and is not billed.
mode string "" Set to auto to let webfetch pick the cheapest configuration that works, as ScrapingBee's Auto-Mode does. It tries these configurations in order and stops at the first that succeeds: no rendering (1 credit), render_js (5), premium_proxy without rendering (10), premium_proxy with rendering (25), then stealth_proxy on a datacenter exit and then on an ISP exit (billed as stealth_proxy is). Do not send render_js, premium_proxy, stealth_proxy or transparent_status_code with it: auto mode chooses them, and sending one is a 400. A 404, 410, 413, a robots refusal or a blocked address is a real answer and is never retried. Every other non-2xx target status, a timeout, a failed render, or an anti-bot challenge or CAPTCHA page (whatever status it came with) moves on to the next configuration. Configurations that failed cost nothing, so you are billed once, for the one that worked, reported in Wf-Auto-Cost (0 when none did). The configurations tried come back in Wf-Auto-Attempts.
max_cost integer "" Ceiling in credits on a single mode=auto call: only configurations that cost at most this much are tried, so the charge never exceeds it. max_cost=5 tries a raw fetch and a rendered one but never a proxy; max_cost=25 climbs up to premium_proxy with rendering but never reaches stealth_proxy. A whole number of at least 1. Ignored without mode=auto.
transparent_status_code boolean false Return the target's status code verbatim instead of ours.
tag string "" Your own label, echoed back in Wf-Tag.
respect_robots boolean true Webfetch only, paid plans only. Setting false needs a paid plan and the robots.txt opt-out turned on for the key in the key portal (off by default; it records your acceptance of the acceptable use policy), and every such request is logged. A free key is refused with 403 robots_optout_paid_only, and a paid key without the opt-out with 403 aup_required.

An unrecognized parameter does not fail the request. It comes back listed in Wf-Unknown-Parameters so a typo is visible rather than silent.

Credits

ConfigurationCredits
No JavaScript rendering1
JavaScript rendering5
Premium proxy10
Premium proxy with rendering25
Stealth proxy75
Any failed request0

Identical to ScrapingBee's table. Every response carries Wf-Cost with what was charged. If you ask for a proxy pool this deployment cannot provide, the request is served over the best route available, billed for the route it got, and marked Wf-Proxy-Downgraded: true.

extract_rules

A JSON object mapping a name to a selector. The response is JSON in that shape instead of HTML.

{"title": "h1", "link": "a@href"}

Expands to the full form, which you can also write out:

{
  "title": {"selector": "h1", "output": "text", "type": "item"},
  "link":  {"selector": "a",  "output": "@href", "type": "item"}
}
FieldValuesDefault
selectorA CSS selector or an XPath expressionrequired
selector_typeauto, css, xpath. Under auto, a leading / means XPath.auto
typeitem for the first match, list for all of themitem
outputtext, html, @attribute, table_json, table_array, text_relevant, markdown_relevant, or a nested object of further rulestext
cleanCollapse runs of whitespace and trimtrue

Nested rules run inside each matched element, and can mix CSS and XPath:

{
  "articles": {
    "selector": ".card",
    "type": "list",
    "output": {
      "title": ".post-title",
      "link":  {"selector": ".post-title a", "output": "@href"},
      "tags":  {"selector": ".tag", "type": "list"}
    }
  }
}

A whole table, keyed by its header row:

{"pricing": {"selector": "#pricing", "output": "table_json"}}

[{"Plan": "Starter", "Price": "$7.99"}, {"Plan": "Pro", "Price": "$19.99"}]

A selector that matches nothing yields null, or an empty array for a list. One bad rule never fails the whole extraction.

js_scenario

Actions performed in the page before it is captured. Requires render_js=true and completes within 40 seconds.

{
  "strict": true,
  "instructions": [
    {"wait_for": ".results"},
    {"click": "li.next > a"},
    {"wait": 1500},
    {"evaluate": "document.querySelectorAll('.item').length"}
  ]
}
InstructionArgument
clickselector
waitmilliseconds
wait_forselector
wait_for_and_clickselector
scroll_x, scroll_ypixels
fill[selector, value]
evaluateJavaScript; results land in evaluate_results
infinite_scroll{max_count, delay, end_click: {selector}}

strict defaults to true and stops on the first failing step. Set it false to continue. With json_response=true the response carries a js_scenario_report with per-step timings and outcomes.

Response

By default the body is exactly what the target returned. Metadata travels in headers, and the target's own headers are mirrored with a Wf- prefix.

HeaderMeaning
Wf-CostCredits charged, 0 for anything not billed
Wf-ChallengeOn a challenge_page error: the anti-bot vendor recognized (for example cloudflare, datadome)
Wf-Initial-Status-CodeThe status the target returned
Wf-Resolved-UrlThe URL after redirects
Wf-Proxy-PoolRoute used: direct, datacenter, isp, residential, mobile or own
Wf-Proxy-DowngradedPresent when a requested pool was unavailable; the call was billed for the route it got (see Wf-Cost)
Wf-Screenshot-UrlLink to the capture, when one was requested
Wf-TagYour tag, echoed back
Wf-RateLimit-*Limit (the monthly allowance, or on the free tier the 1,000-credit balance cap), Remaining (credits left, bonus included), Resource, and Reset (Unix time of the next reset: a paid plan's renewal date, or the free tier's top-up on the 1st)

With json_response=true everything arrives in one object:

{
  "headers": { ... },
  "cost": 5,
  "initial-status-code": 200,
  "resolved-url": "https://example.com/",
  "type": "html",
  "body": "<html>...</html>",
  "cookies": [ ... ],
  "evaluate_results": [ ... ],
  "iframes": [ ... ],
  "xhr": [ ... ],
  "metadata": {"json-ld": [ ... ], "microdata": [ ... ]}
}

Status codes

By default we report our own status, not the target's: any target 2xx becomes 200, 404, 410 and 413 pass through, and everything else becomes 500. Redirects are followed first, so the status is the end of the chain. Set transparent_status_code=true to get the target's code verbatim.

An anti-bot challenge, CAPTCHA or block page is never billed, whatever status it arrived with, including the ones served as 200. It comes back as an error with code challenge_page, and mode=auto moves on to the next configuration. On the free plan, an account's first such error can also carry a stealth_preview_url. Fetching it, within 10 minutes, runs the page once in our Stealth browser without billing it, and answers with a JPEG of the top of the page (the rest blurred) when Stealth got through, or 204 No Content when it did not.

CodeBilledMeaning
200yesSuccess
400noA parameter is missing or invalid
401noInvalid key, or out of credits
403noTarget blocked, redirected somewhere blocked, refused by robots.txt (code robots_disallowed), or a site our proxy network will not connect to, such as a bank or payment processor (code target_restricted; see the network page)
404 / 410yesThe target says the URL is gone
413yesThe target refused the request as too large
422noA js_scenario step failed with strict on (code js_scenario_failed). The body carries js_scenario_report with the step that failed. Sending it again gives the same answer; fix the step, or set strict to false to get the page anyway
429noRate limited, or too many concurrent requests
500noWe could not fetch it, the target returned a status we do not pass through, or it answered with an anti-bot challenge, CAPTCHA or block page instead of the content (code challenge_page, vendor in Wf-Challenge). When our proxy could not connect to the target, the code says why: target_unreachable, dns_failed, timeout or fetch_failed. A proxy's own error page is never passed on as the target's answer; we try another exit first
503noTry again shortly. On the free tier, paid traffic has priority (code free_tier_busy; retry after the Retry-After seconds). On any plan, our fetch network could not complete the request this time (code fleet_unavailable, egress_unavailable, robots_unavailable or screenshot_unavailable)

Every error body is JSON with error, code and docs. The code is stable and worth branching on.

Usage endpoint

curl "https://api.webfetch.io/v1/usage" -H "Authorization: Bearer wfk_your_key"
{
  "plan": "starter",
  "max_api_credit": 75000,
  "used_api_credit": 1284,
  "bonus_credits": 0,
  "max_concurrency": 25,
  "current_concurrency": 0,
  "renewal_subscription_date": "2026-11-10T10:00:00+00:00"
}

On a paid plan, used_api_credit counts the current billing period and renewal_subscription_date is the renewal date, when the allowance resets. On a free account, max_api_credit is the balance plus what was used this month, so max_api_credit - used_api_credit is what is left, as a ScrapingBee client computes it. The response also carries free_credit_balance, free_credit_monthly_topup and free_credit_cap, and renewal_subscription_date is when the next top-up lands, at 00:00 UTC on the 1st.

Ranked search results from Google, Bing, Yahoo and Brave as JSON, in the shape of ScrapingBee's Google Search API: the same parameter names and defaults, the same response keys and the same credit costs. ScrapingBee has no Bing, Yahoo or Brave endpoint, so those take the Google parameters and return the same shape.

curl "https://api.webfetch.io/v1/google?search=python+asyncio+tutorial" \
  -H "Authorization: Bearer wfk_your_key"
EngineEndpointTry it
Google Search APIGET https://api.webfetch.io/v1/googleGoogle search tool
Bing Search APIGET https://api.webfetch.io/v1/bingBing search tool
Yahoo Search APIGET https://api.webfetch.io/v1/yahooYahoo search tool
Brave Search APIGET https://api.webfetch.io/v1/braveBrave search tool

The same paths answer under https://webfetch.io/api/v1/, and with ScrapingBee's older store/ segment (/api/v1/store/google). GET only.

Who can use it

Search needs the robots.txt opt-out turned on for the key. Every engine's robots.txt disallows automated access to its result pages, and webfetch honors robots.txt until the opt-out is on, which records your acceptance of the acceptable use policy. Paid plans turn it on in the key portal. Free accounts request search access there (name, country and what you will search for), and a person approves each request; approved free keys search one at a time, and respect_robots=false on the scraping API stays on paid plans. A free key without approval gets 403 search_access_required, or 403 search_access_pending while its request is reviewed; a key without the opt-out gets 403 aup_required. Every search is logged as a robots.txt opt-out request.

Parameters

ParameterTypeDefaultDescription
searchstringrequiredThe query, as typed in the search box. Operators such as site: and intitle: work.
country_codestringusTwo-letter country the search is run from. A country no search exit can leave from is refused with 400 country_unavailable.
languagestringenInterface language, such as de. Applies to Bing and Google; Yahoo follows the exit's location.
pageinteger1Results page, 1 to 50. Bing and Google show 10 results a page, Yahoo 7.
light_requestbooleantrue10 credits when true, 15 when false. See below for what changes per engine.
search_typestringclassicclassic for the full results page, or ads for the ads only. Same price either way.
add_htmlbooleanfalseAlso return the page's HTML, in html.
tagstring""Your own label, echoed back in Wf-Tag.
nfprbooleanfalseGoogle only. Turn off Google's spelling auto-correction.
extra_paramsstring""Google only. Extra Google URL parameters, URL-encoded, limited to ScrapingBee's list: filter, fpstate, locale, nfpr, safe, safe_search, tbm, tbs, udm. Any other key is a 400. tbm (news, images, videos, shopping) and any udm other than 14 (the Web filter) switch Google to a layout the endpoint does not read yet, and are a 400 too.

nb_results is accepted for compatibility and has no effect. Not supported yet, each refused with a 400 naming it: device=mobile, pages above 1, date_range, latitude, longitude, radius, sort_by, min_price, max_price, and search types other than classic and ads.

Credits

RequestCredits
Light request (light_request=true, the default)10
Regular request (light_request=false)15
Any failed search0

Every engine costs the same, and so does search_type=ads. A results page is billed even when it has no results, because that is the engine's answer. A block, a timeout or a page that is not a results page is never billed. On Bing, a light request is a plain HTTP fetch of the first page: about half a second, with the organic results and no ads, because Bing shows its ads only to a browser. Regular requests, ads-only requests and pages after the first are loaded in a browser. Google, Yahoo and Brave always load in a browser; a light request skips images, fonts and media.

Response

Every key is always present, a list with nothing in it is an empty array, and positions start at 1.

{
  "meta_data": {
    "url": "https://www.bing.com/search?q=python%20asyncio%20tutorial&setlang=en&cc=US",
    "number_of_results": 2500,
    "location": null,
    "number_of_organic_results": 10,
    "number_of_ads": 0,
    "number_of_page": 1,
    "zero_results_for_original_query": false
  },
  "organic_results": [
    {
      "position": 1,
      "title": "Python's asyncio: A Hands-On Walkthrough - Real Python",
      "url": "https://realpython.com/async-io-python/",
      "displayed_url": "https://realpython.com \u203a async-io-python",
      "description": "In this tutorial, you'll learn how Python asyncio works ...",
      "domain": "realpython.com",
      "date": "Jul 30, 2025",
      "date_utc": null,
      "rich_snippet": {},
      "sitelinks": []
    },
    ...
  ],
  "top_ads": [],
  "bottom_ads": [],
  "related_queries": [{"position": 1, "title": "async python documentation", "url": "https://www.bing.com/search?q=..."}],
  "related_searches": [{"position": 1, "query": "async python documentation", "link": "https://www.bing.com/search?q=...", "type": "standard"}],
  "questions": [{"position": 1, "text": "What is asyncio in Python?", "answer": null}],
  "knowledge_graph": {},
  "local_results": [], "top_stories": [], "news_results": [], "images": [],
  "map_results": [], "hotel_results": [], "ai_overviews": []
}

url is the destination itself; the engines' click-tracking redirects are decoded, and a result whose destination cannot be resolved is left out rather than returned as an engine link (the Wf-Unresolved-Results header counts them). displayed_url is the address as the engine shows it, or null where it shows something else there (Google shows counts such as "10+ answers" for some sites), and date the date it shows beside the snippet, when there is one.

zero_results_for_original_query is true when the search found nothing for the query as sent. A quoted phrase asks for an exact match: when an engine drops the quotes and answers a looser query (no result shows the phrase, or even its longest word; all four do it), organic_results is empty rather than filled with results for another search. When Google says it searched for a correction ("Showing results for"), its results are kept and the flag is set; send nfpr=1 to search the query exactly as written.

Ads only

With search_type=ads the response has the same keys, with the ads filled in and every other list empty. Each ad has its position in its block, its landing page and the engine's click URL. A page that shows no ads returns empty lists.

"top_ads": [
  {
    "position": 1,
    "title": "5 Best Coffee Makers | Our #1 Pick Will Surprise You",
    "url": "https://bestreviews.com/best-coffee-makers?...",
    "tracking_url": "https://www.bing.com/aclk?ld=...",
    "visual_url": "https://bestreviews.com \u203a coffee-makers",
    "domain": "bestreviews.com",
    "description": "Looking for the Best Coffee Maker on the Market? ...",
    "snippet": null,
    "sitelinks": []
  }
],
"bottom_ads": [ ... ]

domain is the advertiser's displayed domain. url can be the advertiser's own click tracker when that is where the ad points.

Headers and errors

Responses carry Wf-Cost, Wf-Resolved-Url (the results page loaded), Wf-Proxy-Pool, Wf-Tag and the Wf-RateLimit-* headers. A search blocked on one exit is retried on another, up to three times within the 90-second limit; when that happened, Wf-Search-Attempts says how many tries it took.

CodeBilledMeaning
200yesA results page, including one with no results (an empty organic_results and zero_results_for_original_query set to true)
400noA parameter is missing, invalid or not supported yet (bad_request), or the country has no search exit (country_unavailable)
401noNo key, an invalid key, out of credits, or a closed account
403noSearch is off for this key: search_access_required or search_access_pending (free plan, not approved yet) or aup_required (opt-out not turned on)
429noRate limited, or too many concurrent requests for the key
500noThe engine answered with a bot check on every try (challenge_page), the page could not be loaded or was not a results page (search_failed), or it ran out of time (timeout)
503noSearch capacity is busy (search_busy, with Retry-After), or the engine is not open yet (engine_unavailable)

Official SDKs

MIT licensed, and thin: each wraps the same endpoint documented above. Source is on GitHub.

LanguageInstallRequires
JavaScriptnpm install @tuxxin/webfetchNode 18+, no runtime dependencies
Pythonpip install webfetch-ioPython 3.9+, sync and async, httpx; imports as webfetch
PHPcomposer require tuxxin/webfetchPHP 8.1+, cURL directly
import { WebfetchClient } from '@tuxxin/webfetch';
const client = new WebfetchClient(process.env.WEBFETCH_KEY);

const md = await client.markdown('https://example.com');

const data = await client.extract('https://quotes.toscrape.com/js/', {
  quotes: { selector: '.quote', type: 'list',
            output: { text: '.text', author: '.author' } },
});

Errors raise the same three levels everywhere: WebfetchError, then WebfetchApiError with the status and a stable code, then WebfetchRateLimitError carrying the retry delay (a 429, or a 503 when free capacity is busy). A failed call is never billed, so there is nothing to reconcile afterwards.

MCP server

Webfetch speaks the Model Context Protocol, so an MCP-aware client can call it as a tool rather than scraping this page.

{
  "mcpServers": {
    "webfetch": {
      "url": "https://webfetch.io/mcp",
      "headers": { "Authorization": "Bearer wfk_your_key" }
    }
  }
}
ToolArguments
fetch_pageurl, render_js, return_format (markdown, text or html)
extract_dataurl, extract_rules, render_js

JSON-RPC 2.0 over HTTP POST, protocol version 2025-06-18. A key is required, sent as Authorization: Bearer wfk_... on the HTTP request; calls draw on the key's credits (the free balance, or a paid plan's monthly allowance). Every result reports the credits it cost. The server card is at /.well-known/mcp/server-card.json. The endpoint is POST only and answers 405 to GET.

Migrating from ScrapingBee

  1. Change the base URL from https://app.scrapingbee.com/api/v1/ to https://api.webfetch.io/v1/.
  2. Swap the key.
  3. If you read Spb- response headers, read Wf- instead. Forwarded request headers still accept the Spb- prefix, so those need no change.

Parameter names, defaults, the credit table, the JSON envelope keys and the status-code rules are all the same. Two things are not, and both change what a request does rather than what it is called.

The proxy parameters map to different pools. ScrapingBee's premium_proxy is residential; ours routes static ISP proxies, addresses registered to internet providers that sites trust more than datacenter IPs. Their stealth_proxy is a separate pool with anti-bot hardening (they do not publish the proxy type); ours runs a hardened browser on a datacenter exit first, then an ISP exit if the site refuses, and says which in Wf-Proxy-Pool and Wf-Proxy-Downgraded. See the comparison.

Not implemented: the AI extraction parameters (ai_query, ai_extract_rules, ai_selector) and scraping_config. Sending one is reported in Wf-Unknown-Parameters rather than silently ignored. ScrapingBee's Google Search endpoint is at the same path here, /api/v1/google; see Search API.

Limits and fair use

  • Concurrent requests are capped per plan: 5 on free, 25 on Starter, up to 400 on Business+. Custom is bounded only by the hardware it runs on.
  • The free tier is a balance: 1,000 credits at signup, then 250 more on the 1st of each month, never topped up past 1,000. Unused free credits carry over up to that cap; they do not carry over into a paid plan. A paid plan's allowance resets each month on its renewal date, the day PayPal bills it.
  • Bonus credits from a promo code are one-time. They are spent only after the free balance or monthly allowance is used up, so they survive a reset. The remaining balance is reported in Wf-Bonus-Credits and by /v1/usage.
  • Requests to a single target host are capped per key and across all callers, so nobody can point the service at one victim.
  • robots.txt is honored by default. Paid plans can opt out; see the acceptable use policy first. The search endpoints need the opt-out, because every engine's robots.txt disallows its result pages.
  • Private, loopback, link-local and reserved addresses are refused, and the check is repeated on every redirect.
  • Each scrape has a bandwidth ceiling, measured on the wire rather than on the returned document, so assets a page loads count toward it: 5 MB through the stealth pools (mobile and residential) and 25 MB everywhere else. A page that reaches the ceiling still returns the content captured up to that point, billed normally, marked Wf-Truncated: true with the ceiling in Wf-Max-Bytes. Ordinary pages are nowhere near it: across a sample of major news, reference and retail sites the median rendered scrape moved under 1 MB.