A hidden API is an undocumented endpoint that a website's own front end calls to load data, usually returning JSON. It is not hidden in a security sense, since the browser requests it in plain view, but it has no public documentation or stability promise.
Many JavaScript-heavy pages render from JSON fetched after the first load. Open the browser's Network panel, filter to Fetch/XHR, then trigger the action you care about, such as scrolling, sorting, or changing a page number. The request that returns the data you see on screen is the candidate, and copying it as a cURL command preserves its headers, cookies, and body for replay.
Check the page source first. Some sites embed the same data as JSON inside a script tag in the initial HTML, which needs no extra request at all. Both routes skip layout markup, and the JSON can include fields the page never displays, such as internal identifiers.
A hidden endpoint has no versioning contract. Parameter names, response shapes, and required headers can change with any front-end deploy, so a scraper built on one needs monitoring and a fallback. Endpoints often require a Referer, a specific Accept header, or a short-lived token generated by page scripts, and replaying them without that context returns errors.
They can also sit behind the same bot protection as the page, so a direct call from a datacenter IP may be challenged even though the browser call succeeded. Check the site's terms and robots.txt before relying on an undocumented endpoint, because being technically reachable is not the same as being permitted.
import requests
url = "https://example.com/api/products"
headers = {
"Accept": "application/json",
"Referer": "https://example.com/products",
"User-Agent": "Mozilla/5.0 (compatible; example-scraper)",
}
resp = requests.get(url, params={"page": 1}, headers=headers, timeout=15)
resp.raise_for_status()
for item in resp.json().get("items", []):
print(item.get("id"), item.get("name"))How Crawlora handles this
Crawlora exists partly so you do not have to maintain these reverse-engineered calls. For supported platforms, its REST endpoints return structured JSON with the proxying and browser rendering handled for you, billed on successful requests. Check the platform catalog for your target before building your own endpoint scraper.
Related reading
Glossary
FAQ
It depends on the site's terms, the data involved, and your jurisdiction. Being reachable from a browser does not grant permission, so review the terms of service and robots.txt, and avoid personal data you have no basis to collect.
The endpoint may require tokens, cookies, or headers that page scripts generate, or it may sit behind bot protection that fingerprints the client. Compare your request to the browser's, including header order and cookies.
Often cheaper and faster, because JSON needs no layout parsing. But the endpoint is undocumented and can change without notice, so parsed HTML or a maintained API can be more stable for long-running jobs.
Beyond Hidden API (Internal JSON Endpoint), Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.