Documentation
One endpoint. Send a URL and get back rendered HTML, clean text, markdown, a screenshot, or JSON shaped by your own selectors. A browser and a rotating proxy pool sit behind it. Already integrated against ScrapingBee? Parameter names, defaults and credit costs match theirs, so it's a base URL and a key. See Migrating from ScrapingBee.
Quick start
Every request needs a key. A free account starts with 1,000 credits, gets 250 more every month (banking up to 1,000), and needs only an email address.
curl "https://api.webfetch.io/v1/?url=https%3A%2F%2Fexample.com&render_js=false" \
-H "Authorization: Bearer wfk_your_key"Asking for structured data instead of HTML:
curl "https://api.webfetch.io/v1/" \
-H "Authorization: Bearer wfk_your_key" \
--data-urlencode "url=https://quotes.toscrape.com/js/" \
--data-urlencode 'extract_rules={"quotes":{"selector":".quote","type":"list","output":{"text":".text","author":".author"}}}' \
-GBoth hosts serve the same API:
https://api.webfetch.io/v1/
https://webfetch.io/api/v1/Authentication
Send your key as a bearer token:
Authorization: Bearer wfk_your_key
The api_key query parameter also works, because ScrapingBee accepts
it and migrating code often uses it. It is the weaker option: query strings end up
in logs and referrers, so prefer the header.
There is no keyless tier. Rate limiting alone does not hold against a large
address pool, and an anonymous endpoint leaves no way to act against whoever
abuses it. A request without a key returns 401 with code
key_required.
Parameters
Sent as a query string on GET, or as a JSON or form body on POST. Query parameters still apply on a POST.
| Parameter | Type | Default | Description |
|---|---|---|---|
api_key |
string | required |
Your key. Prefer the Authorization: Bearer header; this query parameter is accepted for compatibility. |
url |
string | required |
The URL to fetch, URL-encoded. |
render_js |
boolean | true |
Render in a browser. Set false for a plain HTTP fetch at 1 credit instead of 5. |
wait |
integer ms | 0 |
Extra settle time after load. Maximum 35000. |
wait_for |
string | "" |
CSS or XPath selector to wait for before capturing. |
wait_browser |
string | domcontentloaded |
One of domcontentloaded, load, networkidle0, networkidle2. |
js_scenario |
stringified JSON | {} |
Click, fill, scroll and wait before capture. See below. |
window_width |
integer | 1920 |
Viewport width, 200 to 3840. |
window_height |
integer | 1080 |
Viewport height, 200 to 2160. |
premium_proxy |
boolean | false |
Route through static ISP proxies: addresses registered to internet providers rather than hosting companies, which sites trust more than datacenter IPs. 10 credits, or 25 with rendering. With no ISP proxy available the call falls back to the residential pool (same price), then a datacenter IP at the base rate, and says so in Wf-Proxy-Downgraded and Wf-Proxy-Pool. Note for ScrapingBee users: their premium_proxy is residential. |
stealth_proxy |
boolean | false |
For targets that refuse ordinary traffic. Renders in our stealth browser, built to get past bot checks: it runs the page's scripts with real mouse and scroll input and loads every resource, takes its time zone and language from the exit, and costs 75 credits. A request it cannot take (with cookies, forwarded headers, js_scenario, wait_for, block_resources=true sent explicitly, block_ads, device=mobile, another window size, return_page_source or screenshot_selector) renders in our hardened standard browser instead, billed for the exit used. It goes out on a datacenter exit first; if the site refuses it (an error, a block status or a challenge page), it is tried once more on an ISP exit. Implies render_js. Billed for the exit that answered when the stealth browser could not take it: the base rate (5) on datacenter, the premium rate (25) on ISP or residential, 75 on a mobile exit. Wf-Proxy-Pool names the pool used and Wf-Proxy-Downgraded: true marks a call billed below 75, so you are never charged for more than you got. Capped at 5 MB per scrape on the mobile and residential pools. |
country_code |
string | "" |
Two-letter ISO 3166-1 code for the exit country. A plain request exits from the datacenter pool's countries at no extra cost; with premium_proxy or stealth_proxy it uses that pool's countries (see the network page). A country the route cannot exit from is refused with 400 country_unavailable rather than served from another country. |
own_proxy |
string | "" |
Use your own proxy: [http://]<user>:<pass>@<host>:<port>. Port defaults to 1080. Only HTTP proxies are supported right now: https and socks proxies are refused with 400 own_proxy_unsupported. |
session_id |
integer | "" |
Route requests sharing this id through the same exit IP. |
block_resources |
boolean | true |
Skip images, media, fonts and stylesheets, and strip third-party telemetry: ad exchanges, analytics, tag managers, error reporting, consent banners and social embed widgets. Measured across major news and retail sites this removes about 37% of the bytes while preserving 100% of the text, headings and links, because none of it carries page content. The main document is never blocked whatever its host, and the apex domains customers scrape are never on the list. Set false to load the page exactly as a browser would. |
block_ads |
boolean | false |
Strip the same telemetry and ad hosts even when block_resources is off. Redundant at the default, kept because it is a ScrapingBee parameter. |
extract_rules |
stringified JSON | "" |
CSS or XPath selectors to structured JSON. See below. |
return_page_source |
boolean | false |
Return the HTML as delivered, before JavaScript ran. |
return_page_text |
boolean | false |
Return readable text with menus, headers and footers removed. |
return_page_markdown |
boolean | false |
Return the content as markdown, with the navigation, header, footer and banners removed and links made absolute. See URL to Markdown. |
json_response |
boolean | false |
Wrap everything in a JSON envelope instead of returning the body raw. |
add_html |
boolean | false |
With json_response=true and a transformation (extract_rules, return_page_text or return_page_markdown), also include the untransformed HTML in body. A request with no transformation already returns it either way. |
screenshot |
boolean | false |
Also capture a screenshot. Returned as a link, at no extra credit cost. Not available together with js_scenario or stealth_proxy right now (400 screenshot_unsupported). |
screenshot_full_page |
boolean | false |
Capture the whole scrollable page. |
screenshot_selector |
string | "" |
Report the bounding box of this selector with the screenshot. |
cookies |
string | "" |
Cookies to send: name=value,domain=example.com;other=value2. |
forward_headers |
boolean | false |
Forward headers you prefix with Wf- (or Spb-). |
forward_headers_pure |
boolean | false |
Forward only your headers, adding none of ours. |
device |
string | desktop |
desktop or mobile. |
timeout |
integer ms | 140000 |
Budget for the fetch, as at ScrapingBee. Whatever it says, a call is answered within 90 seconds in total, mode=auto routes included, because the network edge in front of the API closes longer connections. A request that runs out of time returns a timeout error and is not billed. |
mode |
string | "" |
Set to auto to let webfetch pick the cheapest configuration that works, as ScrapingBee's Auto-Mode does. It tries these configurations in order and stops at the first that succeeds: no rendering (1 credit), render_js (5), premium_proxy without rendering (10), premium_proxy with rendering (25), then stealth_proxy on a datacenter exit and then on an ISP exit (billed as stealth_proxy is). Do not send render_js, premium_proxy, stealth_proxy or transparent_status_code with it: auto mode chooses them, and sending one is a 400. A 404, 410, 413, a robots refusal or a blocked address is a real answer and is never retried. Every other non-2xx target status, a timeout, a failed render, or an anti-bot challenge or CAPTCHA page (whatever status it came with) moves on to the next configuration. Configurations that failed cost nothing, so you are billed once, for the one that worked, reported in Wf-Auto-Cost (0 when none did). The configurations tried come back in Wf-Auto-Attempts. |
max_cost |
integer | "" |
Ceiling in credits on a single mode=auto call: only configurations that cost at most this much are tried, so the charge never exceeds it. max_cost=5 tries a raw fetch and a rendered one but never a proxy; max_cost=25 climbs up to premium_proxy with rendering but never reaches stealth_proxy. A whole number of at least 1. Ignored without mode=auto. |
transparent_status_code |
boolean | false |
Return the target's status code verbatim instead of ours. |
tag |
string | "" |
Your own label, echoed back in Wf-Tag. |
respect_robots |
boolean | true |
Webfetch only, paid plans only. Setting false needs a paid plan and the robots.txt opt-out turned on for the key in the key portal (off by default; it records your acceptance of the acceptable use policy), and every such request is logged. A free key is refused with 403 robots_optout_paid_only, and a paid key without the opt-out with 403 aup_required. |
An unrecognized parameter does not fail the request. It comes back listed in
Wf-Unknown-Parameters so a typo is visible rather than silent.
Credits
| Configuration | Credits |
|---|---|
| No JavaScript rendering | 1 |
| JavaScript rendering | 5 |
| Premium proxy | 10 |
| Premium proxy with rendering | 25 |
| Stealth proxy | 75 |
| Any failed request | 0 |
Identical to ScrapingBee's table. Every response carries Wf-Cost with
what was charged. If you ask for a proxy pool this deployment cannot
provide, the request is served over the best route available, billed for the route
it got, and marked Wf-Proxy-Downgraded: true.
extract_rules
A JSON object mapping a name to a selector. The response is JSON in that shape instead of HTML.
{"title": "h1", "link": "a@href"}Expands to the full form, which you can also write out:
{
"title": {"selector": "h1", "output": "text", "type": "item"},
"link": {"selector": "a", "output": "@href", "type": "item"}
}| Field | Values | Default |
|---|---|---|
selector | A CSS selector or an XPath expression | required |
selector_type | auto, css, xpath. Under auto, a leading / means XPath. | auto |
type | item for the first match, list for all of them | item |
output | text, html, @attribute, table_json, table_array, text_relevant, markdown_relevant, or a nested object of further rules | text |
clean | Collapse runs of whitespace and trim | true |
Nested rules run inside each matched element, and can mix CSS and XPath:
{
"articles": {
"selector": ".card",
"type": "list",
"output": {
"title": ".post-title",
"link": {"selector": ".post-title a", "output": "@href"},
"tags": {"selector": ".tag", "type": "list"}
}
}
}A whole table, keyed by its header row:
{"pricing": {"selector": "#pricing", "output": "table_json"}}
[{"Plan": "Starter", "Price": "$7.99"}, {"Plan": "Pro", "Price": "$19.99"}]A selector that matches nothing yields null, or an empty
array for a list. One bad rule never fails the whole extraction.
js_scenario
Actions performed in the page before it is captured. Requires
render_js=true and completes within 40 seconds.
{
"strict": true,
"instructions": [
{"wait_for": ".results"},
{"click": "li.next > a"},
{"wait": 1500},
{"evaluate": "document.querySelectorAll('.item').length"}
]
}| Instruction | Argument |
|---|---|
click | selector |
wait | milliseconds |
wait_for | selector |
wait_for_and_click | selector |
scroll_x, scroll_y | pixels |
fill | [selector, value] |
evaluate | JavaScript; results land in evaluate_results |
infinite_scroll | {max_count, delay, end_click: {selector}} |
strict defaults to true and stops on the first failing
step. Set it false to continue. With json_response=true the response
carries a js_scenario_report with per-step timings and outcomes.
Response
By default the body is exactly what the target returned. Metadata travels in
headers, and the target's own headers are mirrored with a Wf-
prefix.
| Header | Meaning |
|---|---|
Wf-Cost | Credits charged, 0 for anything not billed |
Wf-Challenge | On a challenge_page error: the anti-bot vendor recognized (for example cloudflare, datadome) |
Wf-Initial-Status-Code | The status the target returned |
Wf-Resolved-Url | The URL after redirects |
Wf-Proxy-Pool | Route used: direct, datacenter, isp, residential, mobile or own |
Wf-Proxy-Downgraded | Present when a requested pool was unavailable; the call was billed for the route it got (see Wf-Cost) |
Wf-Screenshot-Url | Link to the capture, when one was requested |
Wf-Tag | Your tag, echoed back |
Wf-RateLimit-* | Limit (the monthly allowance, or on the free tier the
1,000-credit balance cap), Remaining (credits left, bonus included),
Resource, and Reset (Unix time of the next reset: a paid plan's renewal date, or the free tier's top-up on the 1st) |
With json_response=true everything arrives in one object:
{
"headers": { ... },
"cost": 5,
"initial-status-code": 200,
"resolved-url": "https://example.com/",
"type": "html",
"body": "<html>...</html>",
"cookies": [ ... ],
"evaluate_results": [ ... ],
"iframes": [ ... ],
"xhr": [ ... ],
"metadata": {"json-ld": [ ... ], "microdata": [ ... ]}
}Status codes
By default we report our own status, not the target's: any target 2xx becomes
200, 404, 410 and 413 pass through, and everything else becomes 500. Redirects
are followed first, so the status is the end of the chain. Set
transparent_status_code=true to get the target's code verbatim.
An anti-bot challenge, CAPTCHA or block page is never billed, whatever status it
arrived with, including the ones served as 200. It comes back as an error with code
challenge_page, and mode=auto moves on to the next
configuration. On the free plan, an account's first such error can also carry a
stealth_preview_url. Fetching it, within 10 minutes, runs the page once in
our Stealth browser without billing it, and answers with a JPEG of the top of the page
(the rest blurred) when Stealth got through, or 204 No Content when it did not.
| Code | Billed | Meaning |
|---|---|---|
| 200 | yes | Success |
| 400 | no | A parameter is missing or invalid |
| 401 | no | Invalid key, or out of credits |
| 403 | no | Target blocked, redirected somewhere blocked, refused by robots.txt (code robots_disallowed), or a site our proxy network will not connect to, such as a bank or payment processor (code target_restricted; see the network page) |
| 404 / 410 | yes | The target says the URL is gone |
| 413 | yes | The target refused the request as too large |
| 422 | no | A js_scenario step failed with strict on (code js_scenario_failed). The body carries js_scenario_report with the step that failed. Sending it again gives the same answer; fix the step, or set strict to false to get the page anyway |
| 429 | no | Rate limited, or too many concurrent requests |
| 500 | no | We could not fetch it, the target returned a status we do not pass through, or it answered with an anti-bot challenge, CAPTCHA or block page instead of the content (code challenge_page, vendor in Wf-Challenge). When our proxy could not connect to the target, the code says why: target_unreachable, dns_failed, timeout or fetch_failed. A proxy's own error page is never passed on as the target's answer; we try another exit first |
| 503 | no | Try again shortly. On the free tier, paid traffic has priority (code free_tier_busy; retry after the Retry-After seconds). On any plan, our fetch network could not complete the request this time (code fleet_unavailable, egress_unavailable, robots_unavailable or screenshot_unavailable) |
Every error body is JSON with error, code
and docs. The code is stable and worth branching on.
Usage endpoint
curl "https://api.webfetch.io/v1/usage" -H "Authorization: Bearer wfk_your_key"{
"plan": "starter",
"max_api_credit": 75000,
"used_api_credit": 1284,
"bonus_credits": 0,
"max_concurrency": 25,
"current_concurrency": 0,
"renewal_subscription_date": "2026-11-10T10:00:00+00:00"
}On a paid plan, used_api_credit counts the current billing period and
renewal_subscription_date is the renewal date, when the allowance resets.
On a free account, max_api_credit is the balance plus what was
used this month, so max_api_credit - used_api_credit is what is left, as a
ScrapingBee client computes it. The response also carries free_credit_balance,
free_credit_monthly_topup and free_credit_cap, and
renewal_subscription_date is when the next top-up lands, at 00:00 UTC on the 1st.
Search API
Ranked search results from Google, Bing, Yahoo and Brave as JSON, in the shape of ScrapingBee's Google Search API: the same parameter names and defaults, the same response keys and the same credit costs. ScrapingBee has no Bing, Yahoo or Brave endpoint, so those take the Google parameters and return the same shape.
curl "https://api.webfetch.io/v1/google?search=python+asyncio+tutorial" \
-H "Authorization: Bearer wfk_your_key"| Engine | Endpoint | Try it |
|---|---|---|
| Google Search API | GET https://api.webfetch.io/v1/google | Google search tool |
| Bing Search API | GET https://api.webfetch.io/v1/bing | Bing search tool |
| Yahoo Search API | GET https://api.webfetch.io/v1/yahoo | Yahoo search tool |
| Brave Search API | GET https://api.webfetch.io/v1/brave | Brave search tool |
The same paths answer under https://webfetch.io/api/v1/, and with ScrapingBee's
older store/ segment (/api/v1/store/google). GET only.
Who can use it
Search needs the robots.txt opt-out turned on for the key. Every engine's robots.txt disallows
automated access to its result pages, and webfetch honors robots.txt until the opt-out is on, which
records your acceptance of the acceptable use policy. Paid plans turn it on in the
key portal. Free accounts request search access there (name, country and what you
will search for), and a person approves each request; approved free keys search one at a time, and
respect_robots=false on the scraping API stays on paid plans. A free key without approval
gets 403 search_access_required, or 403 search_access_pending while its
request is reviewed; a key without the opt-out gets 403 aup_required. Every search is
logged as a robots.txt opt-out request.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
search | string | required | The query, as typed in the search box. Operators such as site: and intitle: work. |
country_code | string | us | Two-letter country the search is run from. A country no search exit can leave from is refused with 400 country_unavailable. |
language | string | en | Interface language, such as de. Applies to Bing and Google; Yahoo follows the exit's location. |
page | integer | 1 | Results page, 1 to 50. Bing and Google show 10 results a page, Yahoo 7. |
light_request | boolean | true | 10 credits when true, 15 when false. See below for what changes per engine. |
search_type | string | classic | classic for the full results page, or ads for the ads only. Same price either way. |
add_html | boolean | false | Also return the page's HTML, in html. |
tag | string | "" | Your own label, echoed back in Wf-Tag. |
nfpr | boolean | false | Google only. Turn off Google's spelling auto-correction. |
extra_params | string | "" | Google only. Extra Google URL parameters, URL-encoded, limited to ScrapingBee's list: filter, fpstate, locale, nfpr, safe, safe_search, tbm, tbs, udm. Any other key is a 400. tbm (news, images, videos, shopping) and any udm other than 14 (the Web filter) switch Google to a layout the endpoint does not read yet, and are a 400 too. |
nb_results is accepted for compatibility and has no effect. Not supported yet,
each refused with a 400 naming it: device=mobile, pages above 1,
date_range, latitude, longitude, radius,
sort_by, min_price, max_price, and search types other than
classic and ads.
Credits
| Request | Credits |
|---|---|
Light request (light_request=true, the default) | 10 |
Regular request (light_request=false) | 15 |
| Any failed search | 0 |
Every engine costs the same, and so does search_type=ads. A results page is billed
even when it has no results, because that is the engine's answer. A block, a timeout or a page
that is not a results page is never billed.
On Bing, a light request is a plain HTTP fetch of the first page: about half a second, with the
organic results and no ads, because Bing shows its ads only to a browser. Regular requests,
ads-only requests and pages after the first are loaded in a browser.
Google, Yahoo and Brave always load in a browser;
a light request skips images, fonts and media.
Response
Every key is always present, a list with nothing in it is an empty array, and positions start at 1.
{
"meta_data": {
"url": "https://www.bing.com/search?q=python%20asyncio%20tutorial&setlang=en&cc=US",
"number_of_results": 2500,
"location": null,
"number_of_organic_results": 10,
"number_of_ads": 0,
"number_of_page": 1,
"zero_results_for_original_query": false
},
"organic_results": [
{
"position": 1,
"title": "Python's asyncio: A Hands-On Walkthrough - Real Python",
"url": "https://realpython.com/async-io-python/",
"displayed_url": "https://realpython.com \u203a async-io-python",
"description": "In this tutorial, you'll learn how Python asyncio works ...",
"domain": "realpython.com",
"date": "Jul 30, 2025",
"date_utc": null,
"rich_snippet": {},
"sitelinks": []
},
...
],
"top_ads": [],
"bottom_ads": [],
"related_queries": [{"position": 1, "title": "async python documentation", "url": "https://www.bing.com/search?q=..."}],
"related_searches": [{"position": 1, "query": "async python documentation", "link": "https://www.bing.com/search?q=...", "type": "standard"}],
"questions": [{"position": 1, "text": "What is asyncio in Python?", "answer": null}],
"knowledge_graph": {},
"local_results": [], "top_stories": [], "news_results": [], "images": [],
"map_results": [], "hotel_results": [], "ai_overviews": []
}
url is the destination itself; the engines' click-tracking redirects are decoded,
and a result whose destination cannot be resolved is left out rather than returned as an
engine link (the Wf-Unresolved-Results header counts them).
displayed_url is the address as the engine shows it, or null where it
shows something else there (Google shows counts such as "10+ answers" for some sites), and
date the date it shows beside the snippet, when there is one.
zero_results_for_original_query is true when the search found nothing
for the query as sent. A quoted phrase asks for an exact match: when an engine drops the quotes
and answers a looser query (no result shows the phrase, or even its longest word; all four do it),
organic_results is empty rather than filled with results for another search. When
Google says it searched for a correction ("Showing results for"), its results are kept and the
flag is set; send nfpr=1 to search the query exactly as written.
Ads only
With search_type=ads the response has the same keys, with the ads filled in and
every other list empty. Each ad has its position in its block, its landing page and the
engine's click URL. A page that shows no ads returns empty lists.
"top_ads": [
{
"position": 1,
"title": "5 Best Coffee Makers | Our #1 Pick Will Surprise You",
"url": "https://bestreviews.com/best-coffee-makers?...",
"tracking_url": "https://www.bing.com/aclk?ld=...",
"visual_url": "https://bestreviews.com \u203a coffee-makers",
"domain": "bestreviews.com",
"description": "Looking for the Best Coffee Maker on the Market? ...",
"snippet": null,
"sitelinks": []
}
],
"bottom_ads": [ ... ]
domain is the advertiser's displayed domain. url can be the
advertiser's own click tracker when that is where the ad points.
Headers and errors
Responses carry Wf-Cost, Wf-Resolved-Url (the results page loaded),
Wf-Proxy-Pool, Wf-Tag and the Wf-RateLimit-* headers.
A search blocked on one exit is retried on another, up to three times within the
90-second limit; when that happened, Wf-Search-Attempts says how many tries it took.
| Code | Billed | Meaning |
|---|---|---|
| 200 | yes | A results page, including one with no results (an empty organic_results and zero_results_for_original_query set to true) |
| 400 | no | A parameter is missing, invalid or not supported yet (bad_request), or the country has no search exit (country_unavailable) |
| 401 | no | No key, an invalid key, out of credits, or a closed account |
| 403 | no | Search is off for this key: search_access_required or search_access_pending (free plan, not approved yet) or aup_required (opt-out not turned on) |
| 429 | no | Rate limited, or too many concurrent requests for the key |
| 500 | no | The engine answered with a bot check on every try (challenge_page), the page could not be loaded or was not a results page (search_failed), or it ran out of time (timeout) |
| 503 | no | Search capacity is busy (search_busy, with Retry-After), or the engine is not open yet (engine_unavailable) |
Official SDKs
MIT licensed, and thin: each wraps the same endpoint documented above. Source is on GitHub.
| Language | Install | Requires |
|---|---|---|
| JavaScript | npm install @tuxxin/webfetch | Node 18+, no runtime dependencies |
| Python | pip install webfetch-io | Python 3.9+, sync and async, httpx; imports as webfetch |
| PHP | composer require tuxxin/webfetch | PHP 8.1+, cURL directly |
import { WebfetchClient } from '@tuxxin/webfetch';
const client = new WebfetchClient(process.env.WEBFETCH_KEY);
const md = await client.markdown('https://example.com');
const data = await client.extract('https://quotes.toscrape.com/js/', {
quotes: { selector: '.quote', type: 'list',
output: { text: '.text', author: '.author' } },
});
Errors raise the same three levels everywhere: WebfetchError,
then WebfetchApiError with the status and a stable code, then
WebfetchRateLimitError carrying the retry delay (a 429, or a 503 when
free capacity is busy). A failed call is never billed, so there is nothing to
reconcile afterwards.
MCP server
Webfetch speaks the Model Context Protocol, so an MCP-aware client can call it as a tool rather than scraping this page.
{
"mcpServers": {
"webfetch": {
"url": "https://webfetch.io/mcp",
"headers": { "Authorization": "Bearer wfk_your_key" }
}
}
}| Tool | Arguments |
|---|---|
fetch_page | url, render_js, return_format (markdown, text or html) |
extract_data | url, extract_rules, render_js |
JSON-RPC 2.0 over HTTP POST, protocol version 2025-06-18. A key is
required, sent as Authorization: Bearer wfk_... on the HTTP request; calls
draw on the key's credits (the free balance, or a paid plan's monthly allowance).
Every result reports the credits it cost. The
server card is at /.well-known/mcp/server-card.json. The endpoint
is POST only and answers 405 to GET.
Migrating from ScrapingBee
- Change the base URL from
https://app.scrapingbee.com/api/v1/tohttps://api.webfetch.io/v1/. - Swap the key.
- If you read
Spb-response headers, readWf-instead. Forwarded request headers still accept theSpb-prefix, so those need no change.
Parameter names, defaults, the credit table, the JSON envelope keys and the status-code rules are all the same. Two things are not, and both change what a request does rather than what it is called.
The proxy parameters map to different pools.
ScrapingBee's premium_proxy is residential; ours routes static ISP
proxies, addresses registered to internet providers that sites trust more than
datacenter IPs. Their stealth_proxy is a separate pool with anti-bot
hardening (they do not publish the proxy type); ours runs a hardened browser on a
datacenter exit first, then an ISP exit if the site refuses, and says which in
Wf-Proxy-Pool and Wf-Proxy-Downgraded.
See the comparison.
Not implemented: the AI extraction parameters (ai_query,
ai_extract_rules, ai_selector) and
scraping_config. Sending one is reported in
Wf-Unknown-Parameters rather than silently ignored.
ScrapingBee's Google Search endpoint is at the same path here,
/api/v1/google; see Search API.
Limits and fair use
- Concurrent requests are capped per plan: 5 on free, 25 on Starter, up to 400 on Business+. Custom is bounded only by the hardware it runs on.
- The free tier is a balance: 1,000 credits at signup, then 250 more on the 1st of each month, never topped up past 1,000. Unused free credits carry over up to that cap; they do not carry over into a paid plan. A paid plan's allowance resets each month on its renewal date, the day PayPal bills it.
- Bonus credits from a promo code are one-time. They are spent only after the
free balance or monthly allowance is used up, so they survive a reset. The
remaining balance is reported in
Wf-Bonus-Creditsand by/v1/usage. - Requests to a single target host are capped per key and across all callers, so nobody can point the service at one victim.
robots.txtis honored by default. Paid plans can opt out; see the acceptable use policy first. The search endpoints need the opt-out, because every engine's robots.txt disallows its result pages.- Private, loopback, link-local and reserved addresses are refused, and the check is repeated on every redirect.
- Each scrape has a bandwidth ceiling, measured on the wire rather than on
the returned document, so assets a page loads count toward it:
5 MB through the stealth pools (mobile and residential) and
25 MB everywhere else. A page that reaches the ceiling
still returns the content captured up to that point, billed normally,
marked
Wf-Truncated: truewith the ceiling inWf-Max-Bytes. Ordinary pages are nowhere near it: across a sample of major news, reference and retail sites the median rendered scrape moved under 1 MB.