Website scraper API · Tool
Web Crawling API
Follow links from one page across a site and get every page's clean Markdown, with depth, scope patterns and robots.txt rules.
- 1 credit per call
- Failed calls are free
- Updated
- web.crawl
Response
completed
{ "completion_reason": "page_limit", "final_url": "https://docs.stripe.com/payments", "traversal_complete": false, "pages": [ { "title": "Payments | Stripe Documentation", "content_type": "text/html; charset=utf-8", "depth": 0, "description": "Use Stripe to start accepting payments.", "final_url": "https://docs.stripe.com/payments", "language": "en-US", "markdown": "# Payments | Stripe Documentation\n\nPayments\n\n## Use Stripe to start accepting payments.\n\nGet started\n\nIntegrate with Stripe to start accept…", "rendered": false, "status_code": 200, "url": "https://docs.stripe.com/payments" }, { "title": "Stripe Documentation", "content_type": "text/html; charset=utf-8", "depth": 1, "description": "Explore our guides and examples to integrate Stripe.", "final_url": "https://docs.stripe.com/", "language": "en-US", "markdown": "# Stripe Documentation\n\n\n\n[Sign in](https://dashb…", "parent_url": "https://docs.stripe.com/payments", "rendered": false, "status_code": 200, "url": "https://docs.stripe.com/" } ]}
| # | Title | URL | Depth | Description | Language | Content type | Markdown | Rendered | Status code | Final url |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Payments | Stripe Documentation | https://docs.stripe.com/payments | 0 | Use Stripe to start accepting payments. | en-US | text/html; charset=utf-8 | # Payments | Stripe Documentation Payments ## Use Stripe to sta… | false | 200 | https://docs.stripe.com/payments |
| 2 | Stripe Documentation | https://docs.stripe.com/ | 1 | Explore our guides and examples to integrate Stripe. | en-US | text/html; charset=utf-8 | # Stripe Documentation ![](https://b.stripecdn.com/docs-statics… | false | 200 | https://docs.stripe.com/ |
| 3 | Get started | Stripe Documentation | https://docs.stripe.com/get-started | 1 | Guide to setting up your Stripe account and building your first… | en-US | text/html; charset=utf-8 | # Get started | Stripe Documentation Get started ## Create an a… | false | 200 | https://docs.stripe.com/get-started |
| 4 | Revenue | Stripe Documentation | https://docs.stripe.com/revenue | 1 | Use Stripe's revenue solutions to collect and manage your recur… | en-US | text/html; charset=utf-8 | # Revenue | Stripe Documentation Revenue ## Use Stripe to manag… | false | 200 | https://docs.stripe.com/revenue |
| 5 | Risk | Stripe Documentation | https://docs.stripe.com/risk | 1 | Protect your business against fraud, abuse, and financial losse… | en-US | text/html; charset=utf-8 | # Risk | Stripe Documentation | false | 200 | https://docs.stripe.com/risk |
| 6 | Stripe data | Stripe Documentation | https://docs.stripe.com/data | 1 | Use Stripe data to answer critical questions about the performa… | en-US | text/html; charset=utf-8 | # Stripe data | Stripe Documentation Stripe data ## Use Stripe … | false | 200 | https://docs.stripe.com/data |
| 7 | Money management | Stripe Documentation | https://docs.stripe.com/money-management | 1 | Use Treasury to manage money on Stripe. You can hold funds, sen… | en-US | text/html; charset=utf-8 | # Money management | Stripe Documentation Money management ## S… | false | 200 | https://docs.stripe.com/money-management |
| 8 | Stablecoins and crypto | Stripe Documentation | https://docs.stripe.com/stablecoins | 1 | Accept stablecoin payments, send stablecoin payouts, issue stab… | en-US | text/html; charset=utf-8 | # Stablecoins and crypto | Stripe Documentation Stablecoins and… | false | 200 | https://docs.stripe.com/stablecoins |
| 9 | Embedded finance | Stripe Documentation | https://docs.stripe.com/connect | 1 | Embed payments and financial products in your SaaS platform, ma… | en-US | text/html; charset=utf-8 | # Embedded finance | Stripe Documentation Embedded finance ## E… | false | 200 | https://docs.stripe.com/connect |
| 10 | Developer resources | Stripe Documentation | https://docs.stripe.com/development | 1 | Install the Stripe CLI, set up your development environment for… | en-US | text/html; charset=utf-8 | # Developer resources | Stripe Documentation Developer resource… | false | 200 | https://docs.stripe.com/development |
Input
What you send
The input
object of the request, with every field the endpoint accepts.
| Field | Type | Required | What it does | Example |
|---|---|---|---|---|
| url | string | yes | The page to start from. | https://docs.stripe.com/payments |
| accept_statuses | array<integer> | no | Error statuses you still want returned, such as [404], to inspect the page. | |
| ai | string | no | Rank the documents found by how relevant they are to your instructions. Nothing is removed. | never · auto · always |
| allow_browser_fallback | boolean | no | Open the page in a real browser when it needs JavaScript or blocks a plain request. On by default. | |
| allow_truncated | boolean | no | Return what was read when a page is larger than the size limit, instead of failing. | |
| concurrency | integer | no | How many pages to fetch at once. | |
| continuation_token | string | no | Pass the token from the last response, with the same options, to continue a crawl. | |
| delay_ms | integer | no | A pause between requests, in milliseconds, to go easy on the site. | |
| delivery | string | no | How large results come back: auto, inline, or as a download link for very large crawls. | auto · inline · hosted |
| detail | string | no | summary for titles and counts only; standard or full for each page's text and Markdown. | summary · standard · full |
| document | object | no | Options for PDFs and other files found on the way. | |
| exclude_patterns | array<string> | no | Leave out URLs matching these patterns. | |
| include_patterns | array<string> | no | Only URLs matching these patterns, such as path:/blog/* or contains:pricing. | |
| include_subdomains | boolean | no | Also follow other subdomains of the same site, such as docs. and blog. | |
| instructions | string | no | With ai, which documents you care about. | |
| max_body_bytes | integer | no | The largest page or file to read, in bytes. | |
| max_depth | integer | no | How many links away from the start page to go. | |
| max_pages | integer | no | The most pages to fetch in this call. | 10 |
| max_total_bytes | integer | no | The most bytes to read in all, redirects included. | |
| respect_robots | boolean | no | Follow the site's robots.txt rules. On by default. |
Response
Fields it returns
What each record in the response holds, with the values from the sample above.
| Field | Type | What it is | Example |
|---|---|---|---|
| pages[].title | string | The page's title. | Payments | Stripe Documentation |
| pages[].url | string | The page's address. | https://docs.stripe.com/payments |
| pages[].depth | integer | How many links away from the start page this page is. | 0 |
| pages[].description | string | The page's description for search engines. | Use Stripe to start accepting payments. |
| pages[].language | string | The page's language. | en-US |
| pages[].content_type | string | The kind of content the page is. | text/html; charset=utf-8 |
| pages[].final_url | string | The address the page ended at, after any redirects. | https://docs.stripe.com/payments |
| pages[].markdown | string | The page's text as Markdown. | # Payments | Stripe Documentation Payments ## Use Stripe to sta… |
| pages[].rendered | boolean | Whether the page was opened in a browser before reading. | false |
| pages[].status_code | integer | The site's HTTP status for the page, such as 200. | 200 |
| completion_reason | string | Why the crawl stopped, such as reaching the page limit. | page_limit |
| final_url | string | The address the page ended at, after any redirects. | https://docs.stripe.com/payments |
| pages | array | The pages found. | 10 records |
| traversal_complete | boolean | Whether every reachable page was visited. | false |
Know before you build
What this endpoint covers
- It stays on the site you start from and follows its robots.txt.
- Each page comes back with its Markdown, so one call gives you the text of up to max_pages pages.
- completion_reason says why it stopped: every page reached, or the page limit you set.
- A crawl of many pages can take longer than 30 seconds; fetch the job when it finishes.
REST API · MCP server
Call it from your code, or ask your agent
Set UPSCRAPE_API_KEY to your key and copy a working request.
curl --request POST \ --url https://data.upscrape.com/execute \ --header "Authorization: Bearer $UPSCRAPE_API_KEY" \ --header "Content-Type: application/json" \ --header "Prefer: wait=30" \ --data '{ "capability": "web.crawl", "input": { "max_pages": 10, "url": "https://docs.stripe.com/payments" }}'
import jsonimport osimport requestspayload = json.loads(r'''{ "capability": "web.crawl", "input": { "max_pages": 10, "url": "https://docs.stripe.com/payments" }}''')response = requests.post( "https://data.upscrape.com/execute", headers={ "Authorization": f"Bearer {os.environ['UPSCRAPE_API_KEY']}", "Content-Type": "application/json", "Prefer": "wait=30", }, json=payload,)response.raise_for_status()print(json.dumps(response.json(), indent=2))
const response = await fetch("https://data.upscrape.com/execute", { method: "POST", headers: { "Authorization": `Bearer ${process.env.UPSCRAPE_API_KEY}`, "Content-Type": "application/json", "Prefer": "wait=30", }, body: JSON.stringify({ "capability": "web.crawl", "input": { "max_pages": 10, "url": "https://docs.stripe.com/payments" }}),});if (!response.ok) { throw new Error(`Upscrape request failed: ${response.status}`);}console.log(JSON.stringify(await response.json(), null, 2));
In your AI agent
Claude · Cursor · any MCP client
1Add the Upscrape MCP server
2Ask for the data in plain words
The agent finds the endpoint, shows you the price, and runs it with your credits.
https://data.upscrape.com/mcp
With Upscrape, crawl docs.stripe.com/payments for up to 10 pages and list each page's title and URL.
Website Scraper API
The other website tools
Questions
Web Crawling API FAQ
The short answers. The docs have the long ones.
Still have a question?
Talk to the people who build the endpoints.
How much does the Web Crawling API cost?
1 credit per successful call, shown before it runs. You buy credits once, and paid credits never expire.
What do I send?
Send url. The input fields below list every option with its type and limits.
Which sites does it work on?
Any public page a visitor can open without logging in. A page that is blocked or gone fails with its reason.
Can my AI agent call it?
Yes, over MCP: the agent finds web.crawl, checks its price and runs it with the same credits as the REST API.
Get started
Your first page is free.
Up to 100 free credits to start, no card needed.