Website scraper API · Tool
Sitemap Extractor API
List the URLs a site publishes, from its sitemaps and its pages, filtered by pattern, with titles on request.
- 1 credit per call
- Failed calls are free
- Updated
- web.sitemap
Response
completed
{ "final_url": "https://stripe.com", "origin": "https://stripe.com", "robots": { "applied": true, "status": "applied", "status_code": 200, "url": "https://stripe.com/robots.txt", "sitemaps": [ "https://stripe.com/sitemap/sitemap.xml" ] }, "sitemaps": [ { "note": "no sitemap at this URL (HTTP 404)", "status": "missing", "status_code": 404, "url": "https://stripe.com/sitemap.xml" }, { "status": "ok", "status_code": 200, "url": "https://stripe.com/sitemap/partition-0.xml", "url_count": 1986 } ], "urls": [ { "title": "Stripe | Financial Infrastructure to Grow Your Revenue", "allowed": true, "description": "Stripe is a financial services platform that helps all types of businesses accept payments, build flexible billing models, and manage money…", "kind": "page", "source": "seed", "url": "https://stripe.com/" }, { "allowed": true, "depth": 1, "kind": "page", "source": "page", "url": "https://stripe.com/about" } ], "warnings": [ "mapped_inventory_limit_reached" ]}
| # | URL | Title | Kind | Found in | Allowed |
|---|---|---|---|---|---|
| 1 | https://stripe.com/ | Stripe | Financial Infrastructure to Grow Your Revenue | page | seed | true |
| 2 | https://stripe.com/about | page | page | true | |
| 3 | https://stripe.com/ae | page | page | true | |
| 4 | https://stripe.com/ae/atlas/legal | page | sitemap_alternate | true | |
| 5 | https://stripe.com/ae/climate/legal | page | sitemap_alternate | true | |
| 6 | https://stripe.com/ae/connect-account/legal | page | sitemap_alternate | true |
Input
What you send
The input
object of the request, with every field the endpoint accepts.
| Field | Type | Required | What it does | Example |
|---|---|---|---|---|
| url | string | yes | The site to list, such as https://react.dev/. | https://stripe.com |
| cursor | string | no | Pass the cursor from the last response, with the same options, to get the next URLs. | |
| discover_sitemaps | boolean | no | Look for sitemaps beyond the ones robots.txt names. | |
| exclude_patterns | array<string> | no | Leave out URLs matching these patterns. | |
| fetch_titles | boolean | no | Open each listed page to fill in its title and description. Slower; off by default. | |
| include_assets | boolean | no | Also list scripts, styles, fonts, images and files. Off by default: pages only. | |
| include_patterns | array<string> | no | Only URLs matching these patterns, such as path:/blog/* or contains:pricing. | |
| include_subdomains | boolean | no | Also follow other subdomains of the same site, such as docs. and blog. | |
| max_sitemaps | integer | no | The most sitemap files to read. | |
| max_urls | integer | no | The most URLs to return in this response. | 20 |
| respect_robots | boolean | no | Follow the site's robots.txt rules. On by default. |
Response
Fields it returns
What each record in the response holds, with the values from the sample above.
| Field | Type | What it is | Example |
|---|---|---|---|
| urls[].url | string | The page's address. | https://stripe.com/ |
| urls[].title | string | The page's title, when the site's sitemap or the page gives one. | Stripe | Financial Infrastructure to Grow Your Revenue |
| urls[].kind | string | What the URL is: a page, or an asset when you ask for assets. | page |
| urls[].source | string | Where it was found: the start page, a link on a page, or a sitemap. | seed |
| urls[].allowed | boolean | Whether the site's robots.txt allows reading it. | true |
| urls[].description | string | The page's description, when the site gives one. | Stripe is a financial services platform that helps all types of… |
| final_url | string | The address the page ended at, after any redirects. | https://stripe.com |
| origin | string | The site the map covers. | https://stripe.com |
| robots | object | The page's instructions for search engines. | 5 fields |
| robots.applied | boolean | Whether the site's robots.txt rules were followed. | true |
| robots.sitemaps | array | Sitemaps found for the site. | https://stripe.com/sitemap/sitemap.xml |
| robots.status | string | Whether the page was read. | applied |
| robots.status_code | integer | The site's HTTP status for the page, such as 200. | 200 |
| robots.url | string | The page's address. | https://stripe.com/robots.txt |
| sitemaps | array | Sitemaps found for the site. | 3 records |
| sitemaps[].note | string | Why a sitemap could not be read, such as a missing file. | no sitemap at this URL (HTTP 404) |
| sitemaps[].status | string | Whether the page was read. | missing |
| sitemaps[].status_code | integer | The site's HTTP status for the page, such as 200. | 404 |
| sitemaps[].url | string | The page's address. | https://stripe.com/sitemap.xml |
| urls | array | Every URL found, with its title and description when the site gives them. | 6 records |
| warnings | array | Anything that limited the list, such as the URL limit you set being reached. | mapped_inventory_limit_reached |
Know before you build
What this endpoint covers
- It reads the site's sitemaps and robots.txt, and follows links when they are not enough.
- Titles and descriptions come from the sitemap; set fetch_titles to open each page for them, which is slower.
- A site with more URLs than max_urls returns a continuation token: send it back as cursor for the next batch.
- Pages only by default; set include_assets for scripts, styles, images and files.
REST API · MCP server
Call it from your code, or ask your agent
Set UPSCRAPE_API_KEY to your key and copy a working request.
curl --request POST \ --url https://data.upscrape.com/execute \ --header "Authorization: Bearer $UPSCRAPE_API_KEY" \ --header "Content-Type: application/json" \ --header "Prefer: wait=30" \ --data '{ "capability": "web.sitemap", "input": { "max_urls": 20, "url": "https://stripe.com" }}'
import jsonimport osimport requestspayload = json.loads(r'''{ "capability": "web.sitemap", "input": { "max_urls": 20, "url": "https://stripe.com" }}''')response = requests.post( "https://data.upscrape.com/execute", headers={ "Authorization": f"Bearer {os.environ['UPSCRAPE_API_KEY']}", "Content-Type": "application/json", "Prefer": "wait=30", }, json=payload,)response.raise_for_status()print(json.dumps(response.json(), indent=2))
const response = await fetch("https://data.upscrape.com/execute", { method: "POST", headers: { "Authorization": `Bearer ${process.env.UPSCRAPE_API_KEY}`, "Content-Type": "application/json", "Prefer": "wait=30", }, body: JSON.stringify({ "capability": "web.sitemap", "input": { "max_urls": 20, "url": "https://stripe.com" }}),});if (!response.ok) { throw new Error(`Upscrape request failed: ${response.status}`);}console.log(JSON.stringify(await response.json(), null, 2));
In your AI agent
Claude · Cursor · any MCP client
1Add the Upscrape MCP server
2Ask for the data in plain words
The agent finds the endpoint, shows you the price, and runs it with your credits.
https://data.upscrape.com/mcp
Use Upscrape to list every page react.dev publishes, with titles where it has them.
Website Scraper API
The other website tools
Questions
Sitemap Extractor API FAQ
The short answers. The docs have the long ones.
Still have a question?
Talk to the people who build the endpoints.
How much does the Sitemap Extractor API cost?
1 credit per successful call, shown before it runs. You buy credits once, and paid credits never expire.
What do I send?
Send url. The input fields below list every option with its type and limits.
Which sites does it work on?
Any public page a visitor can open without logging in. A page that is blocked or gone fails with its reason.
Can my AI agent call it?
Yes, over MCP: the agent finds web.sitemap, checks its price and runs it with the same credits as the REST API.
Get started
Your first page is free.
Up to 100 free credits to start, no card needed.