Python Web Scraping
Reading layout

Finding Hidden API Endpoints in Network Traffic

This is the hands-on companion to Reverse-Engineering Private APIs in Python, focused on the one skill that unlocks everything else: spotting the hidden JSON endpoint inside a flood of network requests.

Filtering the Network panel to find a JSON endpoint A mixed list of network requests — image, font, script, and a JSON API call — is passed through a Fetch/XHR filter that keeps only the JSON data request, which is then copied as cURL for replay in Python. All requestslogo.png · Imgapp.js · Scriptlistings · Fetchfont.woff2 · Fontstyle.css · CSSFilterFetch/XHR onlylistings → JSONCopy as cURLhttpx replay
Filtering the Network panel to Fetch/XHR isolates the JSON data call from images, fonts, and scripts.

Quick answer: open DevTools, switch to the Network panel, and filter to Fetch/XHR. Clear the log, then trigger the action that loads your data (scroll, click, search) so only the relevant requests appear. Click each one and read the Response tab until you find your JSON; then right-click and Copy as cURL to capture the exact URL, method, and headers. For traffic DevTools cannot see — mobile apps or obfuscated calls — put mitmproxy in the middle and watch the same requests there.

Filtering the Network Panel

The Network panel logs every request the page makes: images, fonts, scripts, stylesheets, and data calls. The data calls are almost always Fetch/XHR, so click that filter first to hide the noise. Three habits make the hunt fast:

  • Clear before you act. Hit the clear (⊘) button, then perform the single action you care about. Now the panel shows only requests caused by that action.
  • Search inside responses. Use the search (magnifying glass) to grep across all response bodies for a value you can see on the page — a price, a product name, an ID. The request that contains it is your endpoint.
  • Sort by size. Data responses are usually the largest Fetch/XHR entries. Sorting by the Size column floats them to the top.

Once you click a candidate, the detail pane shows Headers (URL, method, request headers), Payload (query string or POST body), and Response (the raw JSON). If the Response tab holds the data you want, you found it. This is the same request you will replay with the headers described in Understanding HTTP Requests and Responses.

Copy as cURL, Then Convert to Python

Reproducing every header by hand is error-prone. Instead, right-click the request and choose Copy → Copy as cURL. That command captures the exact method, URL, headers, and body the browser sent. A captured command looks like this:

curl 'https://www.example.com/api/v2/listings?page=1' \
  -H 'accept: application/json' \
  -H 'referer: https://www.example.com/listings' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36' \
  -H 'x-requested-with: XMLHttpRequest'

You can translate that to Python by hand, or paste it into the uncurl helper to generate the request scaffolding automatically.

python -m pip install "httpx>=0.27" uncurl

import httpx

def fetch_listings(url: str) -> dict:
    # Headers copied verbatim from the browser's "Copy as cURL" output.
    headers = {
        "accept": "application/json",
        "referer": "https://www.example.com/listings",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
        "x-requested-with": "XMLHttpRequest",
    }
    with httpx.Client(http2=True, timeout=15) as client:
        response = client.get(url, headers=headers)
        response.raise_for_status()
        return response.json()


if __name__ == "__main__":
    payload = fetch_listings("https://www.example.com/api/v2/listings?page=1")
    print(f"found {len(payload.get('results', []))} listings")

Keeping the header keys exactly as the browser sent them — including lowercase names — avoids subtle mismatches when a backend inspects specific headers.

When DevTools Is Not Enough: mitmproxy

DevTools only sees traffic from the browser tab it is attached to. Mobile apps, desktop clients, and some heavily obfuscated web calls stay invisible. mitmproxy sits between the client and the internet as an intercepting HTTPS proxy, so every request — wherever it originates — flows through a log you control.

python -m pip install mitmproxy

Start the interactive UI, point your device or browser at the proxy (default port 8080), install mitmproxy's CA certificate so HTTPS can be decrypted, and every request appears in the flow list. To capture programmatically instead of watching by hand, run an addon script:

# save as capture.py, run with:  mitmdump -s capture.py
from mitmproxy import http

INTERESTING = ("/api/", "/graphql", "/v1/", "/v2/")


def response(flow: http.HTTPFlow) -> None:
    url = flow.request.pretty_url
    content_type = flow.response.headers.get("content-type", "")
    if "application/json" in content_type and any(part in url for part in INTERESTING):
        print(f"[{flow.request.method}] {url}")
        print(f"  status: {flow.response.status_code}")
        print(f"  bytes:  {len(flow.response.content)}")

This prints every JSON API call the app makes, filtered to likely endpoints — a fast way to map an interface you cannot see in a browser. For GraphQL traffic, the same technique surfaces the /graphql POST body you then replay following Scraping GraphQL Endpoints.

Edge Cases and Caveats

  • WebSocket data. If the panel shows a ws:// or wss:// connection carrying your data, it is a WebSocket stream, not a request/response call. Inspect the Messages tab; you will need a WebSocket client rather than httpx.
  • Certificate pinning. Some mobile apps refuse mitmproxy's CA and drop the connection. Pinning blocks interception on unrooted devices and often signals the app actively resists inspection.
  • Signed or hashed parameters. A query param that looks like a random hash is usually computed in JavaScript. You must read the site's JS to reproduce it, or drive a real browser that computes it for you.
  • Endpoints behind anti-bot layers. A request that works from the browser but returns 403 from Python may be gated by fingerprinting. That is a separate problem covered in Advanced Scraping Techniques & Anti-Bot Evasion.
  • Preflight noise. OPTIONS requests are CORS preflights, not data calls — ignore them and focus on the GET/POST that follows.

Frequently Asked Questions

The Network tab shows hundreds of requests. How do I narrow it down? Filter to Fetch/XHR, clear the log, then trigger only the action you care about. If several JSON requests still appear, search the responses for a value visible on the page — the request containing it is the one you want.

What is the difference between the Fetch/XHR filter and the "Doc" filter? "Doc" shows the top-level HTML document the browser first loaded. Fetch/XHR shows the background data calls JavaScript makes afterward. Private APIs almost always live under Fetch/XHR.

Do I need mitmproxy if I only scrape websites? Usually not — DevTools sees all browser traffic. Reach for mitmproxy when you target a mobile or desktop app, or when a web call is obfuscated in a way that is easier to observe from a neutral proxy.

Copy as cURL gave me a huge header list. Do I need all of it? No. Start with User-Agent, Accept, and Referer, then add headers back one at a time only if the request fails. Once it succeeds, keep the request lean so it is easier to maintain.

Back to home