Documentation

One endpoint. Send a URL and get back rendered HTML, clean text, markdown, a screenshot, or JSON shaped by your own selectors. Headless Chrome and a rotating proxy pool sit behind it. Already integrated against ScrapingBee? Parameter names, defaults and credit costs match theirs, so it's a base URL and a key. See Migrating from ScrapingBee.

Quick start

Every request needs a key. A free account starts with 1,000 credits, gets 250 more every month (banking up to 1,000), and needs only an email address.

curl "https://api.webfetch.io/v1/?url=https%3A%2F%2Fexample.com&render_js=false" \
  -H "Authorization: Bearer wfk_your_key"

Asking for structured data instead of HTML:

curl "https://api.webfetch.io/v1/" \
  -H "Authorization: Bearer wfk_your_key" \
  --data-urlencode "url=https://quotes.toscrape.com/js/" \
  --data-urlencode 'extract_rules={"quotes":{"selector":".quote","type":"list","output":{"text":".text","author":".author"}}}' \
  -G

Both hosts serve the same API:

https://api.webfetch.io/v1/
https://webfetch.io/api/v1/

Authentication

Send your key as a bearer token:

Authorization: Bearer wfk_your_key

The api_key query parameter also works, because ScrapingBee accepts it and migrating code often uses it. It is the weaker option: query strings end up in logs and referrers, so prefer the header.

There is no keyless tier. Rate limiting alone does not hold against a large address pool, and an anonymous endpoint leaves no way to act against whoever abuses it. A request without a key returns 401 with code key_required.

Parameters

Sent as a query string on GET, or as a JSON or form body on POST. Query parameters still apply on a POST.

An unrecognized parameter does not fail the request. It comes back listed in Wf-Unknown-Parameters so a typo is visible rather than silent.

Credits

Identical to ScrapingBee's table. Every response carries Wf-Cost with what was actually charged. If you ask for a proxy pool this deployment cannot provide, the request is served over the best route available, billed for the route it actually got, and marked Wf-Proxy-Downgraded: true.

extract_rules

A JSON object mapping a name to a selector. The response is JSON in that shape instead of HTML.

{"title": "h1", "link": "a@href"}

Expands to the full form, which you can also write out:

{
  "title": {"selector": "h1", "output": "text", "type": "item"},
  "link":  {"selector": "a",  "output": "@href", "type": "item"}
}

Nested rules run inside each matched element, and can mix CSS and XPath:

{
  "articles": {
    "selector": ".card",
    "type": "list",
    "output": {
      "title": ".post-title",
      "link":  {"selector": ".post-title a", "output": "@href"},
      "tags":  {"selector": ".tag", "type": "list"}
    }
  }
}

A whole table, keyed by its header row:

{"pricing": {"selector": "#pricing", "output": "table_json"}}

[{"Plan": "Starter", "Price": "$7.99"}, {"Plan": "Pro", "Price": "$19.99"}]

A selector that matches nothing yields null, or an empty array for a list. One bad rule never fails the whole extraction.

js_scenario

Actions performed in the page before it is captured. Requires render_js=true and completes within 40 seconds.

{
  "strict": true,
  "instructions": [
    {"wait_for": ".results"},
    {"click": "li.next > a"},
    {"wait": 1500},
    {"evaluate": "document.querySelectorAll('.item').length"}
  ]
}

strict defaults to true and stops on the first failing step. Set it false to continue. With json_response=true the response carries a js_scenario_report with per-step timings and outcomes.

Response

By default the body is exactly what the target returned. Metadata travels in headers, and the target's own headers are mirrored with a Wf- prefix.

With json_response=true everything arrives in one object:

{
  "headers": { ... },
  "cost": 5,
  "initial-status-code": 200,
  "resolved-url": "https://example.com/",
  "type": "html",
  "body": "<html>...</html>",
  "cookies": [ ... ],
  "evaluate_results": [ ... ],
  "iframes": [ ... ],
  "xhr": [ ... ],
  "metadata": {"json-ld": [ ... ], "microdata": [ ... ]}
}

Status codes

By default we report our own status, not the target's: any target 2xx becomes 200, 404, 410 and 413 pass through, and everything else becomes 500. Redirects are followed first, so the status is the end of the chain. Set transparent_status_code=true to get the target's code verbatim.

An anti-bot challenge, CAPTCHA or block page is never billed, whatever status it arrived with, including the ones served as 200. It comes back as an error with code challenge_page, and mode=auto moves on to the next configuration.

Every error body is JSON with error, code and docs. The code is stable and worth branching on.

Usage endpoint

curl "https://api.webfetch.io/v1/usage" -H "Authorization: Bearer wfk_your_key"

{
  "plan": "starter",
  "max_api_credit": 75000,
  "used_api_credit": 1284,
  "bonus_credits": 0,
  "max_concurrency": 25,
  "current_concurrency": 0,
  "renewal_subscription_date": "2026-10-01T00:00:00+00:00"
}

On a free account, max_api_credit is the balance plus what was used this month, so max_api_credit - used_api_credit is what is left, as a ScrapingBee client computes it. The response also carries free_credit_balance, free_credit_monthly_topup and free_credit_cap, and renewal_subscription_date is when the next top-up lands.

Official SDKs

MIT licensed, and thin: each wraps the same endpoint documented above. Source is on GitHub.

import { WebfetchClient } from '@tuxxin/webfetch';
const client = new WebfetchClient(process.env.WEBFETCH_KEY);

const md = await client.markdown('https://example.com');

const data = await client.extract('https://quotes.toscrape.com/js/', {
  quotes: { selector: '.quote', type: 'list',
            output: { text: '.text', author: '.author' } },
});

Errors raise the same three levels everywhere: WebfetchError, then WebfetchApiError with the status and a stable code, then WebfetchRateLimitError carrying the retry delay (a 429, or a 503 when free capacity is busy). A failed call is never billed, so there is nothing to reconcile afterwards.

MCP server

Webfetch speaks the Model Context Protocol, so an MCP-aware client can call it as a tool rather than scraping this page.

{
  "mcpServers": {
    "webfetch": {
      "url": "https://webfetch.io/mcp",
      "headers": { "Authorization": "Bearer wfk_your_key" }
    }
  }
}

JSON-RPC 2.0 over HTTP POST, protocol version 2025-06-18. A key is required, sent as Authorization: Bearer wfk_... on the HTTP request; calls draw on the key's credits (the free balance, or a paid plan's monthly allowance). Every result reports the credits it cost. The server card is at /.well-known/mcp/server-card.json. The endpoint is POST only and answers 405 to GET.

Migrating from ScrapingBee

1. Change the base URL from https://app.scrapingbee.com/api/v1/ to https://api.webfetch.io/v1/.
2. Swap the key.
3. If you read Spb- response headers, read Wf- instead. Forwarded request headers still accept the Spb- prefix, so those need no change.

Parameter names, defaults, the credit table, the JSON envelope keys and the status-code rules are all the same. Two things are not, and both change what a request actually does rather than what it is called.

The proxy parameters map to different pools. ScrapingBee's premium_proxy is residential; ours routes static ISP proxies, addresses registered to internet providers that sites trust more than datacenter IPs. Their stealth_proxy is a separate pool with anti-bot hardening (they do not publish the proxy type); ours routes mobile exits first, then residential or ISP (billed as premium), in a hardened browser, and says which in Wf-Proxy-Pool and Wf-Proxy-Downgraded. See the comparison.

Not implemented: the AI extraction parameters (ai_query, ai_extract_rules, ai_selector), scraping_config, and the Google Search endpoint. Sending one is reported in Wf-Unknown-Parameters rather than silently ignored.

Limits and fair use

• Concurrent requests are capped per plan: 5 on free, 25 on Starter, up to 400 on Business+. Custom is bounded only by the hardware it runs on.
• The free tier is a balance: 1,000 credits at signup, then 250 more on the 1st of each month, never topped up past 1,000. Unused free credits carry over up to that cap; they do not carry over into a paid plan. Paid allowances reset on the 1st.
• Bonus credits from a promo code are one-time. They are spent only after the free balance or monthly allowance is used up, so they survive a month rollover. The remaining balance is reported in Wf-Bonus-Credits and by /v1/usage.
• Requests to a single target host are capped per key and across all callers, so nobody can point the service at one victim.
• robots.txt is honored by default. See the acceptable use policy before opting out.
• Private, loopback, link-local and reserved addresses are refused, and the check is repeated on every redirect.
• Each scrape has a bandwidth ceiling, measured on the wire rather than on the returned document, so assets a page loads count toward it: 5 MB through the stealth pools (mobile and residential) and 25 MB everywhere else. A page that reaches the ceiling still returns the content captured up to that point, billed normally, marked Wf-Truncated: true with the ceiling in Wf-Max-Bytes. Ordinary pages are nowhere near it: across a sample of major news, reference and retail sites the median rendered scrape moved under 1 MB.