Website scraper API · Tool
Website Scraper API Crawl Site API
Crawl site from Website Scraper API, as JSON.
- 1 credit per call
- Failed calls are free
- Updated
- web.site.crawl
Response
No sample response published yet
The inputs it accepts are below and the API reference has the full response schema. New accounts get free credits to run it.
API reference
Input
What you send
The input
object of the request, with every field the endpoint accepts.
| Field | Type | Required | What it does | Example |
|---|---|---|---|---|
| url | string | yes | Public http(s) site URL to crawl. | https://example.com/ |
| accept_statuses | array<integer> | no | ||
| ai | string | no | Optional non-destructive relevance ranking for discovered documents. auto leaves candidates unranked if AI is unavailable, while always requires ranking to succeed. Allowed values: never, auto, always. Default: never. |
never · auto · always |
| allow_browser_fallback | boolean | no | Default: false. |
|
| allow_truncated | boolean | no | Default: false. |
|
| concurrency | integer | no | From 1 to 16. Default: 4. |
2 |
| continuation_token | string | no | permanent failures are not amplified. | |
| delay_ms | integer | no | From 0 to 5000. Default: 0. |
|
| detail | string | no | Allowed values: summary, standard, full. Default: standard. |
standard |
| document | object | no | ||
| exclude_patterns | array<string> | no | ||
| include_patterns | array<string> | no | ||
| include_subdomains | boolean | no | When true, hosts are still limited to the seed's registrable domain; unrelated hosts are never added. Default: false. |
|
| instructions | string | no | Required when ai is auto or always. Describe which documents/files are relevant. | |
| max_body_bytes | integer | no | From 1 to 104857600. | |
| max_depth | integer | no | From 1 to 10. Default: 2. |
2 |
| max_pages | integer | no | Maximum number of result pages to fetch. From 1 to 100. Default: 10. |
5 |
| max_total_bytes | integer | no | From 1 to 104857600. | |
| respect_robots | boolean | no | Default: true. |
REST API · MCP server
Call it from your code, or ask your agent
Set UPSCRAPE_API_KEY to your key and copy a working request.
curl --request POST \ --url https://data.upscrape.com/execute \ --header "Authorization: Bearer $UPSCRAPE_API_KEY" \ --header "Content-Type: application/json" \ --header "Prefer: wait=30" \ --data '{ "capability": "web.site.crawl", "input": { "concurrency": 2, "detail": "standard", "max_depth": 2, "max_pages": 5, "url": "https://example.com/" }}'
import jsonimport osimport requestspayload = json.loads(r'''{ "capability": "web.site.crawl", "input": { "concurrency": 2, "detail": "standard", "max_depth": 2, "max_pages": 5, "url": "https://example.com/" }}''')response = requests.post( "https://data.upscrape.com/execute", headers={ "Authorization": f"Bearer {os.environ['UPSCRAPE_API_KEY']}", "Content-Type": "application/json", "Prefer": "wait=30", }, json=payload,)response.raise_for_status()print(json.dumps(response.json(), indent=2))
const response = await fetch("https://data.upscrape.com/execute", { method: "POST", headers: { "Authorization": `Bearer ${process.env.UPSCRAPE_API_KEY}`, "Content-Type": "application/json", "Prefer": "wait=30", }, body: JSON.stringify({ "capability": "web.site.crawl", "input": { "concurrency": 2, "detail": "standard", "max_depth": 2, "max_pages": 5, "url": "https://example.com/" }}),});if (!response.ok) { throw new Error(`Upscrape request failed: ${response.status}`);}console.log(JSON.stringify(await response.json(), null, 2));
In your AI agent
Claude · Cursor · any MCP client
1Add the Upscrape MCP server
2Ask for the data in plain words
The agent finds the endpoint, shows you the price, and runs it with your credits.
https://data.upscrape.com/mcp
With Upscrape, get the Website Scraper API crawl site for concurrency "2", detail "standard" and max depth "2". Tell me what it costs before you run it, then show the results as a table.
Website Scraper API
The other website tools
Questions
Website Scraper API Crawl Site API FAQ
The short answers. The docs have the long ones.
Still have a question?
Talk to the people who build the endpoints.
How much does the Website Scraper API Crawl Site API cost?
1 credit per successful call, shown before it runs. You buy credits once, and paid credits never expire.
What do I send?
Send url. The input fields below list every option with its type and limits.
Which sites does it work on?
Any public page a visitor can open without logging in. A page that is blocked or gone fails with its reason.
Can my AI agent call it?
Yes, over MCP: the agent finds web.site.crawl, checks its price and runs it with the same credits as the REST API.
Get started
Your first page is free.
Up to 100 free credits to start, no card needed.