Why we built Upscrape: one data layer for apps, agents and the web

"Get product data from this website" sounds like a scraping task.

For one request, it usually is. Find the page or endpoint, fetch it, parse the response, return the fields you need.

Then the requirements change.

Run it again tomorrow. Run it for hundreds of locations. Keep the output usable when the source changes. Recover cleanly when a request fails. Avoid duplicate work when a retry happens after a timeout. Add another platform with a different data model. Make the result available to a backend service. Then let an AI agent discover and use the same capability.

At that point the hard part is no longer making one scraper work. It is turning changing websites and platforms into capabilities that other software can depend on, again and again.

That is the problem Upscrape is built around.

A scraper is easy to mistake for the whole problem

Take a location-aware commerce platform such as Blinkit.

A useful request is not "download this page". It is closer to: search for protein bars at a particular delivery pincode and return the products shown there right now. Location matters. Pagination matters. The fields that matter are product identity, price, MRP, pack size and inventory.

Upscrape exposes that operation as blinkit.search. It takes a query plus either a six-digit pincode or coordinates, and returns product records with fields such as the product ID, price, MRP, inventory and serving merchant.

That already removes work from the caller. But the request is only the visible edge of the system.

If you depend on that data every day, you also care about what happens when a run takes longer than expected, when a connection drops, when the upstream site changes, when a retry could create duplicate work, or when an input that used to work stops matching the source.

Those are infrastructure problems. A script answers "can we extract this data?" A reusable capability has to answer a harder question: "can another system depend on this operation repeatedly, without understanding the source-specific machinery underneath?"

That distinction explains most of how Upscrape is built.

From individual scrapers to reusable capabilities

We do not think every source should be made to look identical.

Blinkit search has location semantics that do not belong in a Reddit post search. googlemaps.search takes a location bias alongside its query. blinkit.product needs a product ID that makes no sense for a generic web page. Erasing those differences produces an abstraction that is either misleading or too vague to be useful.

So Upscrape models source-specific work as capabilities: named operations such as blinkit.search or reddit.posts.search, each with its own documented input and behaviour.

Each capability has a published contract that owns its input schema, examples, timeout and credit cost, and every public platform page is generated from it. The raw output stays open-ended, because upstream sources evolve.

"Structured data" does not have to mean pretending every platform shares one universal schema. It can mean the caller invokes a known operation with validated input and gets JSON back through a consistent execution system, instead of rebuilding the source interaction from raw HTML every time.

For source-specific work there are Platform APIs. For the rest of the public web, Universal Web captures pages, extracts fields you define, maps and crawls sites, and takes screenshots.

The abstraction sits above the source without pretending the source does not matter.

Execution is part of the product

There is a large gap between "the parser returned the right fields once" and "this is an operation another application should depend on". The second needs an execution model.

Upscrape's REST surface has one entry point, POST /execute. A request names a registered capability and supplies input that matches its JSON Schema. Quick work completes inline; longer work returns a durable job that you poll, rather than resubmit.

A shared execution model gives one place to handle the concerns that otherwise get rebuilt around every script:

• Retries without duplicates. An idempotency key makes an uncertain retry resolve to the same logical job instead of starting new work.

• Rate limits with timing. A rate-limited request says how long to wait before trying again.

• Stable error codes. Failures carry machine-readable codes, so application logic never parses error prose (errors and retries).

• Pending work you can find again. A job ID survives a timeout on your side.

This is what the first of those looks like in practice. On 23 September 2026 we sent the same blinkit.location request twice with one operation key, the MCP equivalent of Idempotency-Key:

The retry did not run the capability again. The docs add that a replay "does not add another capability charge".

These details are less visible than a scraper demo, but they are closer to the reason the platform exists. The first successful response proves that collection is possible. Durable execution is what makes it useful to other systems.

It also moves where maintenance sits. A team that runs its own scraper owns the source logic and everything around it. With a maintained capability, Upscrape takes on the source-facing work: source access, retries and upstream changes stay behind the endpoint. The consuming team still owns its application logic, storage, analysis and domain checks.

That is a narrower claim than "scraping maintenance disappears". It does not. The point is that every downstream consumer should not have to own the same source maintenance separately.

One execution layer, different consumers

Once a capability exists, the next question is who is calling it.

A backend service and an AI agent may need the same data without wanting the same interface. So we separate the capability and its execution from the way a consumer finds and calls it.

This is what we mean by one data layer.

It does not mean one giant endpoint that hides every useful difference between platforms, and it does not mean every source returns the same schema. It means applications, pipelines, automations and agents can all reach web data without each of them building its own source-specific collection system.

The consumer can change while the collection and execution underneath stay the same.

REST is for software that already knows what it wants

Application code usually benefits from determinism.

If your backend already knows it needs blinkit.search, there is little value in asking a language model to rediscover that choice for every request. The application calls the capability directly, validates its input, keeps the job ID when it needs to, and controls retries, storage, scheduling and processing itself.

That is the role of REST in Upscrape. The REST or MCP guide recommends it when your service knows the capability it wants, or needs explicit control over persistence, batching, scheduling, idempotency, retries or observability.

That stays true in systems full of AI. An agent may write or run software that ends up making ordinary REST calls. Having a model somewhere in the architecture does not make an agent protocol the right runtime interface for every request.

MCP solves a different problem

Agents bring a different integration problem. Instead of code picking a known endpoint, the model may need to work out which capability fits the task, read its inputs, call it and fetch the result.

That is where MCP helps. MCP is an open protocol for connecting AI applications to external tools and data. A server publishes tools; the protocol defines how an AI host discovers and calls them. What a tool does to produce its result is up to the server.

Upscrape's MCP server exposes the same live catalog REST uses. By default it presents four tools, upscrape_search_capabilities, upscrape_describe_capability, upscrape_execute and upscrape_get_job_result, instead of loading hundreds of capability schemas into an agent's context before any of them is needed.

So MCP changes how a capability is discovered and called. It does not replace the collection layer. If an agent asks for location-aware grocery data, MCP helps it find the right capability and build valid input. Getting that data from the source still has to happen somewhere, and so do retries, job state, source maintenance, authorisation, rate handling and delivery.

That is why "MCP versus scraping infrastructure" is the wrong comparison. MCP can put web-data capabilities in front of an agent; it does not make the system underneath unnecessary.

In Upscrape, REST and MCP use the same registered catalog and execution plane. The input schema, account visibility, execution, credit cost and result provenance are the same on both. One execution layer, different consumers.

The browser is an interface, not the data model

A browser is an excellent interface for a person. It is also the right automation tool when a workflow is genuinely interactive: moving through stateful screens, finishing multi-step flows, or taking actions that only exist in the rendered application.

But a browser is not automatically the best contract for software that needs a known piece of data again and again. If a backend needs a list of products, rebuilding that list from a screen designed for people ties the application to details that were never meant for programs. The same goes for agents: a browser helps when the job is browsing, and is a detour when the job is a known structured operation.

The goal is not to get rid of browsers, official APIs or direct integrations. It is to use the interface the workload needs. We work through that choice in detail in web scraping for AI agents.

Where a source offers an official API with the data you need, acceptable terms and suitable limits, that may be the most direct route. Where the work is interactive, browser automation may be right. Where a team has one small, stable internal target, running its own scraper can be entirely reasonable.

Upscrape is for the cases where web data matters enough to use repeatedly, but collecting it is not the part every downstream system should rebuild.

Repeated execution changes the problem again

An on-demand call is not the end state for every workload.

Suppose a team wants a price and availability snapshot every morning, or wants to know when selected fields on a public source change, or needs the same dataset in its warehouse on a schedule. The capability is still needed, but the work now has time and state as well.

Someone has to schedule the runs. Failures have to be told apart from genuine source changes. Repeated runs need bounded retries. Earlier results become the baseline for comparison. Data may need to land in a warehouse or an export rather than in a user-facing response.

You can build that around the API yourself; the Blinkit price-tracker guide does exactly that with Python, SQLite and cron. Or you can let Upscrape run the schedule. Monitors, in private beta, run any published Platform API or Universal Web capability on an interval from one hour to 30 days and compare selected fields of the structured output between runs. Each run goes through the normal job system, not a separate scraping stack.

For work that should not live in a customer's code at all, we also take on custom data projects, where the sources, fields, schedule and delivery are agreed for the job. Get in touch to scope one.

These are different surfaces with the same idea behind them. Once collection has become a reusable capability, you should be able to call it from an application, let an agent use it, or put a schedule around it, without rebuilding the source integration each time.

Reliability is more than an HTTP 200

A successful response does not prove the data is fit for every downstream action.

The input contract matters. Job state matters. The difference between a retry and a duplicate matters. Sources add fields. A technically successful extraction can still contain unexpected or incomplete upstream data.

This matters most for agents. Upscrape's provenance guidance treats every field of capability output as untrusted input. Applications are expected to validate the values they act on, keep retrieved text out of a model's trusted instructions, and ignore unknown fields as sources evolve. Every MCP result carries a _provenance marker saying exactly that.

So a web-data layer should not be mistaken for an oracle. Its job is a cleaner boundary around collection and execution. The consumer still decides what the data means for its product, and how much checking a consequential action needs.

That boundary is part of what makes the model work: collection, execution, application meaning and business decisions stay separate concerns, instead of hiding inside one large "scraper".

Breadth only helps with a common execution model

Adding another scraper to a folder is easy to count. It does not make the system easier to use. As coverage grows, a collection of unrelated scripts creates more integration work, not less: a different way to call each one, different failure handling, different retry logic, different assumptions.

This is how the public catalog looked on 23 September 2026:

The number itself matters less than the fact that every one of those capabilities is registered in one catalog and runs through one execution system, with its platform page generated from the same contract that REST and MCP use.

A new platform still needs source-specific knowledge. It should. But adding it does not create a new integration model for every application and agent that will use it. The source-specific complexity stays source-specific; the execution model is shared.

What we are building

The simplest description of Upscrape is not "a collection of scraping APIs". It is a layer that turns source-specific collection work into reusable capabilities, and runs all of them through one execution model.

Applications and data pipelines call known capabilities over REST. AI agents discover and call the same capabilities over MCP. Recurring work can add a schedule, or become a custom project, when that responsibility belongs outside the consuming application.

The model has limits. Some sources are better reached through their official APIs. Some workflows really need a browser. Some teams should keep a small set of their own scrapers. Some specialised workloads deserve their own infrastructure rather than a general platform.

The argument for Upscrape is not that every web-data problem should become an Upscrape request. It is that once an organisation depends on data from changing websites and platforms, rebuilding collection, execution, retries, contracts and integrations inside every consumer is usually solving the problem at the wrong layer.

The website will keep changing. The application, pipeline, automation or agent that uses its data should not have to change for all the same reasons.