Scraping Mobile App APIs

This guide belongs to Advanced Scraping Techniques and Anti-Bot Evasion, and it covers the surface most people forget a company runs: the backend its phone app talks to. That backend is usually versioned, speaks JSON, ships a stable field set for the lifetime of a release, and carries far less of the anti-automation machinery that guards the public website — which makes it the cleanest thing to read, when you are entitled to read it at all.

The reason is organisational rather than technical. A website is rewritten whenever the design team wants a new look, and it sits behind whatever edge protection the company bought. The mobile backend has to keep working for every app build still installed on a phone somewhere, so it cannot be casually renamed, and it cannot be put behind a JavaScript challenge because an app has no JavaScript engine to solve one. Those two constraints — backwards compatibility and the absence of a browser — push mobile APIs towards being explicit, versioned and script-friendly, which is exactly what a scraper wants.

Before Anything Else: What You Are Entitled To

This page is technical, not legal advice, but the framing matters more here than anywhere else in scraping, so it comes first rather than as a footnote.

Analyse only apps you have obtained legitimately and endpoints you are entitled to call. In practice that means three things. First, use an app build you are allowed to have and a device or emulator image you own — never someone else's handset, and never a corporate device you do not administer. Second, read the terms that govern the service before you send a single automated request; many publish an explicit developer policy, and some publish a real API that makes this whole exercise unnecessary. Third, do not use a captured session to reach data the account you logged in with cannot already see. Replaying your own session at machine speed is a rate-limit question; replaying it against another user's identifiers is not, and this guide does not cover it.

Rate limits deserve the same respect they get on the web. A phone makes a handful of requests when a screen opens; a script can make thousands a minute. If the API publishes limits, stay under them. If it does not, pick a conservative rate and keep it — a mobile backend is often sized for real user traffic, and an unthrottled crawler is easy to notice and trivially easy to block.

Finally, know when to stop. If the host pins its certificate, or the client signs requests with a key you would have to extract from the binary, the intended message is that the endpoint is not for you. The honest options at that point are covered in Handling Certificate Pinning in API Analysis, and the shortest of them is to use the documented API instead.

When to Use a Mobile API

Reach for the app's backend only after cheaper options fail. The checks below run in order, and any one of them can end the exercise.

The first row is the one people skip. Before installing anything, open the site in a browser, watch the network panel, and see whether the page already fetches its data as JSON — the method is written up in Finding Hidden API Endpoints in Network Traffic. If it does, you are done: no emulator, no certificate, no proxy, and the general replay technique in Reverse-Engineering Private APIs applies directly.

The mobile route earns its setup cost in a narrow band of cases: the web is server-rendered and expensive to crawl, the app exposes richer records, or the web layer is defended and the mobile one is not. Outside that band the emulator is overhead.

The "richer records" case is worth expanding, because it is the one that justifies the work most often. A web page is a presentation layer: it shows what the designer chose to show. The mobile endpoint frequently serialises the whole entity, because the app needs the same payload to drive a list view, a detail view and an offline cache. In practice that means internal identifiers the site never prints, stock counts rendered on the web as a vague availability badge, per-variant pricing collapsed into a single "from" figure in HTML, timestamps the page formats away into "3 days ago", and category or tag identifiers that let you join records without string matching. When you compare one JSON record against the page that displays it and the JSON has twice the fields, the app backend is the better source and the setup cost pays for itself in a single crawl.

Prerequisites

You need Python 3.10 or newer, an HTTP client, and an intercepting proxy.

python -m pip install "httpx[http2]>=0.27" "mitmproxy>=10.2" "tenacity>=8.2"

On the device side you need one of the following, all of which you must control yourself:

• An Android emulator image from Android Studio. Choose a Google APIs image rather than a Google Play one — the Play images are production-locked and will not let you modify the system trust store.
• A physical development device that you own and have put into a debuggable state.
• A desktop build of the same app, if the vendor ships one. Electron and macOS builds often speak the identical API and need no emulator at all.

mitmproxy needs no system dependencies beyond Python itself on Linux and macOS; on Windows it is easiest inside WSL. Everything below assumes the proxy runs on your workstation and the device points at it over the local network.

Step-by-Step: Recording and Rebuilding a Mobile API Call

1. Put a Proxy Between the App and the Network

Start the proxy in the mode that writes to a file rather than drawing a terminal interface, so the capture survives the session:

mitmdump --listen-port 8080 --set confdir=~/.mitmproxy -w capture.mitm

Then point the device at it. On an Android emulator the reliable route is a command-line flag at boot rather than the Wi-Fi settings screen, because some system components ignore the UI setting:

emulator -avd Pixel_7_API_34 -http-proxy http://10.0.2.2:8080 -writable-system

10.0.2.2 is the emulator's alias for the host machine's loopback address, and -writable-system is what later lets you place a certificate where the platform will trust it. The full walkthrough — the interface, the filters, and an addon that writes a machine-readable log — is in Intercepting App Traffic with mitmproxy.

The important mental model is in that sequence: the proxy does not decrypt one connection, it terminates one and opens another. The app negotiates TLS with the proxy using a certificate your device has been told to trust, and the proxy negotiates a separate, ordinary TLS session upstream. Plaintext exists only in the middle, inside a process on your own machine.

2. Confirm You Are Seeing Real Traffic

Before reading anything, prove the capture works. Open a screen in the app that clearly loads data and check that flows appear with a method and a path, not just a bare CONNECT line.

mitmdump --listen-port 8080 --set confdir=~/.mitmproxy \
  "~d api.example.com" --set flow_detail=2

Three outcomes are possible, and they mean different things:

• Flows with methods and paths. The capture is working; move on.
• CONNECT lines with nothing inside them. TLS never completed. Either the certificate is not trusted yet, or the host is pinned.
• No flows at all. The traffic is not reaching the proxy — the device is not using it, or the app opens a socket the proxy is not listening on.

The middle case is the common one, and separating "untrusted root" from "pinned host" is a diagnosis, not a guess: an untrusted root fails identically for every host, while a pin fails for one host while the rest of the app keeps working.

3. Read the Request Pattern, Not the Single Request

A capture gives you one call. What you want is the shape all the calls share, because that is what you can generalise into a client. Four things are worth writing down before you touch Python.

Base URL and version segment. Mobile APIs almost always look like https://api.example.com/v3/.... The version is the most useful thing on the page: it tells you the vendor has committed to a stable field set for that number, and it tells you what will change when the shape does — the number, not the field names.

Path grammar. Look at three or four different screens and the resource naming becomes obvious: /v3/catalog/items, /v3/catalog/items/{id}, /v3/catalog/categories. Guessing paths is a bad idea, but recognising a REST convention lets you predict which captured call maps to which screen.

Auth header. Usually a bearer token, occasionally a custom header such as X-Session-Token. Note where it came from — a login call, a refresh call, or an anonymous bootstrap call the app makes on first launch. That last case is common and convenient: many read-only catalogue endpoints are served to a token that any fresh install can obtain.

Device and build headers. A block of headers describing the client: app version, OS version, device model, locale, sometimes a device identifier. They are usually not security controls, but some are load-bearing — a missing X-App-Version occasionally produces a 426 telling you to upgrade.

There is a fifth thing worth capturing that is easy to miss on a first pass: the order in which the app makes its calls when it cold-starts. Kill the app, clear the flow list, and open it fresh. What you usually see is a short, fixed sequence — a configuration or feature-flag call, then an anonymous or refreshed token call, then the first screen's data call. That sequence is the bootstrap, and it is the part you have to reproduce before any interesting request will work. The configuration call in particular often returns the base URL the rest of the session uses, which means hard-coding a host you saw once can break the moment the vendor moves a region behind a different domain.

{
  "method": "GET",
  "url": "https://api.example.com/v3/catalog/items?cursor=eyJwIjoyfQ&limit=40",
  "request_headers": {
    "Host": "api.example.com",
    "Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.example.signature",
    "User-Agent": "ExampleApp/8.4.1 (Android 14; Pixel 7)",
    "X-App-Version": "8.4.1",
    "X-Platform": "android",
    "Accept": "application/json",
    "Accept-Encoding": "gzip"
  },
  "status": 200,
  "response_headers": {
    "Content-Type": "application/json; charset=utf-8",
    "X-RateLimit-Limit": "600",
    "X-RateLimit-Remaining": "597"
  }
}

Two details in that sample are worth more than the rest. X-RateLimit-Limit is a published budget — obey it, and you never have to think about being blocked. And the cursor value is opaque base64: it is a server-side bookmark, not an offset you can compute, which decides the shape of your pagination loop.

4. Know What Makes a Mobile Client Identifiable

A mobile request is easy to tell apart from a browser request, and the differences run in both directions — they help you (the server expects a non-browser client, so no JavaScript challenge is coming) and they expose you (a request that claims to be an app but behaves like httpx is trivially spotted).

The identifying surface has four layers:

1. The User-Agent. App user agents follow their own convention, typically AppName/version (OS version; device model), and often an HTTP library suffix such as okhttp/4.12.0. A request claiming ExampleApp/8.4.1 while sending browser-shaped Accept headers is inconsistent on its face.
2. The header set and its order. Native HTTP stacks send a small, stable, ordered header list. A Python client sends a different set in a different order. Most backends do not check; the ones that do usually check order, not content.
3. The TLS handshake. OkHttp on Android and NSURLSession on iOS produce their own cipher and extension profiles, distinct from both browsers and OpenSSL-based Python clients. If a mobile endpoint fingerprints anything, this is usually it — the mechanics are in TLS and JA3 Fingerprint Evasion.
4. The request rhythm. An app fetches one page, waits for a human to scroll, then fetches the next. A loop with no delay looks nothing like that at the traffic level, regardless of headers.

The practical conclusion is not "spoof everything". It is: send the headers the API actually reads, keep them internally consistent, and pace the requests. Reproducing a device identifier you invented, or a signature you cannot regenerate, creates more failure modes than it removes.

5. Turn the Capture Into a Client

The gap between a captured request and a maintainable client is mostly deletion. A capture contains connection-level headers that your HTTP library must set for itself, a token that expired minutes after it was recorded, and a URL with one specific cursor baked into it. All three have to become something else.

from dataclasses import dataclass

import httpx

BASE_URL = "https://api.example.com/v3"
APP_UA = "ExampleApp/8.4.1 (Android 14; Pixel 7)"


@dataclass(frozen=True)
class Item:
    item_id: str
    name: str
    price: float


def build_client(token: str) -> httpx.Client:
    headers = {
        "User-Agent": APP_UA,
        "Authorization": f"Bearer {token}",
        "X-App-Version": "8.4.1",
        "X-Platform": "android",
        "Accept": "application/json",
    }
    return httpx.Client(base_url=BASE_URL, headers=headers, http2=True, timeout=20.0)


def fetch_page(client: httpx.Client, cursor: str | None) -> tuple[list[Item], str | None]:
    params: dict[str, str | int] = {"limit": 40}
    if cursor is not None:
        params["cursor"] = cursor
    response = client.get("/catalog/items", params=params)
    response.raise_for_status()
    payload = response.json()
    items = [
        Item(item_id=row["id"], name=row["name"], price=float(row["price"]["amount"]))
        for row in payload.get("items", [])
    ]
    return items, payload.get("next_cursor")


if __name__ == "__main__":
    with build_client(token="replace-with-a-token-you-obtained-yourself") as client:
        first_page, next_cursor = fetch_page(client, cursor=None)
        print(f"{len(first_page)} items, next cursor: {next_cursor}")

Note what the client does not do. It does not send Host, Content-Length, Connection or Accept-Encoding — httpx sets all four correctly, and copying them from a capture is how you get mismatched-length errors and undecodable bodies. It does not hard-code a cursor. And it keeps one Client so connections are pooled instead of renegotiating TLS per request. Retry policy, token refresh and typed parsing are built out fully in Replaying Mobile API Requests in Python.

6. Pin the Response Shape With a Fixture

The one maintenance habit worth adopting from day one: save a real response body to disk and write a test that parses it. When the vendor bumps /v3/ to /v4/, or renames price.amount to price.value, the test fails in seconds instead of the crawl quietly producing rows of None.

import json
from pathlib import Path

FIXTURE = Path("fixtures/catalog_items_v3.json")


def load_fixture() -> dict:
    return json.loads(FIXTURE.read_text(encoding="utf-8"))


def test_item_shape_is_stable() -> None:
    payload = load_fixture()
    assert "items" in payload, "top-level items key disappeared"
    first = payload["items"][0]
    for field in ("id", "name", "price"):
        assert field in first, f"item field {field} disappeared"
    assert "amount" in first["price"], "price object changed shape"

Reshaping the nested JSON that mobile APIs love to return — price objects, localised name maps, image arrays — is covered in Parsing JSON and XML Responses, and enforcing types on the way into storage in Validating Scraped Data with Pydantic.

Performance and Scaling Considerations

A mobile endpoint is cheap per record and that is precisely the trap. A single JSON page holding 40 items typically costs 8–40 KB gzipped and one round trip, against several hundred kilobytes and a full render for the equivalent web page. Ten thousand records that would take a browser pool an hour take a single-threaded script a few minutes — which means the limiting factor moves from your machine to the vendor's tolerance almost immediately.

Three numbers should drive your configuration. Page size: take the largest limit the API accepts without erroring, since halving the request count halves your visibility. Concurrency: two to four simultaneous connections is usually plenty; a mobile backend sized for phone traffic rarely thanks you for more, and httpx with HTTP/2 multiplexes several requests over one connection anyway. Delay: if the response carries X-RateLimit-Remaining, use it as a live signal and slow down as it falls rather than waiting for a 429.

Memory is rarely an issue with cursor pagination because you process a page at a time, but it becomes one if you accumulate every record in a list before writing. Stream to disk or a database per page. Token lifetime sets the real ceiling on a long run: a token valid for 60 minutes and a crawl that takes 90 needs refresh logic, not a longer timeout, and a refresh that runs mid-crawl must not lose the cursor it was holding.

For genuinely large jobs the concurrency patterns and backoff policy that apply to any JSON API apply here too — see Retrying Failed Requests with Tenacity for the retry side.

Common Errors and Fixes

Only CONNECT entries appear in the proxy log. TLS never completed, so there is no request to show. Check the trust store first — an untrusted root breaks every host identically. If exactly one host fails while others succeed, that host is pinned.

401 Unauthorized immediately, on a request that worked in the app. The captured token has expired. Mobile access tokens are frequently short-lived, with a separate long-lived refresh token. Re-run the login or refresh call rather than re-capturing by hand.

403 Forbidden with a valid token. Something in the header set is being checked. Add back the app User-Agent, then X-App-Version, then the platform header, one at a time — this is the fastest way to find which one matters, and to discover that most of the others do not.

426 Upgrade Required or a body saying the app version is too old. Your captured X-App-Version has aged out. Update it to the current build number, which is visible in the store listing or in a fresh capture.

httpx.RemoteProtocolError or a body of binary noise. You copied Accept-Encoding or Content-Length from the capture. Delete both and let the client negotiate compression and compute lengths itself.

The loop never terminates. The stop condition is wrong for the pagination shape. Cursor APIs end when the cursor field is null, page APIs when the item list comes back empty, offset APIs when fewer rows arrive than the limit requested. Testing the wrong one either truncates the crawl or spins forever on the last page.

Every request works, but the data is a few hours stale. Mobile endpoints often sit behind a CDN with a short cache lifetime. Look for Age and Cache-Control in the response headers before assuming your parsing is wrong.

A field that was populated last week is now null everywhere. The vendor changed the serialisation without changing the version, which unversioned or loosely versioned endpoints do routinely. This is what the fixture test in step 6 is for; without one, the crawl keeps returning rows and nobody notices for weeks.

404 on a path copied exactly from the capture. Check whether the base URL came from a configuration call rather than being fixed. Region-specific hosts and gradual migrations mean two devices can be given different bases for the same logical endpoint.

Frequently Asked Questions

Is a mobile API easier to scrape than the website? Usually, and for structural reasons rather than negligence. An app has no JavaScript engine, so the vendor cannot put a JavaScript challenge in front of it, and the endpoint must stay compatible with older installed builds, so it cannot be renamed on a whim. The result is a versioned JSON interface that is easier to read and slower to break than the HTML it sits beside.

Do I need a rooted phone to do this? No, and for most work you should not use one. An emulator image you create yourself is easier to reset, easier to script, and does not put a capture certificate on a device you also use personally. Reach for physical hardware only when the app refuses to run on an emulator.

What if the app signs its requests? Then reproducing the call means reproducing a computation that lives inside the binary, and the signing key is deliberately not meant to leave it. That is the vendor stating the endpoint is closed. Treat a signature header as a stop sign and look for a documented API instead.

How long does a captured mobile API keep working? Longer than a set of CSS selectors, typically a full major version of the app — often six to eighteen months. The failure, when it comes, is usually a version bump in the path rather than a renamed field, which is easy to detect if you keep a fixture test that parses a saved response.

Can I use the same approach on a desktop app? Yes. Any client that speaks HTTPS through the system proxy can be recorded the same way, and desktop builds frequently call the identical endpoints with a different User-Agent. When a vendor ships one, it is the lowest-setup option available — no emulator, no system trust store edits.

Related

• Advanced Scraping Techniques and Anti-Bot Evasion — the parent section for this guide
• Intercepting App Traffic with mitmproxy — running the proxy, filtering flows and exporting a capture
• Handling Certificate Pinning in API Analysis — diagnosing a pinned host and the legitimate ways forward
• Replaying Mobile API Requests in Python — headers, tokens, pagination and a full client class
• Reverse-Engineering Private APIs — the same idea applied to a website's own JSON layer