Python Web Scraping
Reading layout

Intercepting App Traffic with mitmproxy

Recording what an app sends is the first practical step in Scraping Mobile App APIs, and mitmproxy is the tool that does it with the least ceremony.

Two TLS segments around an intercepting proxy The app negotiates TLS with the proxy using a locally trusted certificate authority, and the proxy negotiates a separate TLS session with the vendor API. Readable request and response bodies exist only between the two. One connection becomes twoTLS 1: your local CATLS 2: the real chainApptrusts the added CAmitmproxymints a leaf per hostVendor APIsees a normal clientPlaintext exists only inside the proxy processboth segments stay encrypted on the wire; the flow list is what you read
An intercepting proxy does not decrypt one connection — it terminates one and opens another, which is why plaintext exists only inside the proxy process.

The whole job is four commands and one certificate. Start mitmproxy on your workstation, point an emulator you own at it, install the proxy's certificate authority into the emulator's system trust store, and filter the flow list down to the one host you care about. From there a twenty-line addon turns every matching exchange into a line of JSON you can read from Python without ever opening the interface again.

How Interception Actually Works

mitmproxy is not a decryptor. When the app opens a TLS connection to api.example.com, the proxy answers the handshake itself, generating a leaf certificate for that hostname on the fly and signing it with a certificate authority it created on first run. The app validates that certificate against its trust store; if your CA is in there, validation succeeds and the app proceeds as though it were talking to the real server. The proxy then opens a second, entirely ordinary TLS connection upstream.

Two consequences follow, and both are worth internalising before debugging anything.

The first is that trust is the only mechanism at play. Nothing is broken or brute-forced — you added a root to a device you own, and the platform is behaving exactly as designed. Which is also why the CA must never go anywhere near a device you use for anything else: anyone holding that key can do the same thing to your own traffic.

The second is that a client can decline to consult the trust store. An app that ships its own list of acceptable public keys will compare the certificate against that list, find your locally minted one absent, and close the connection. That is certificate pinning, and it is diagnosed in Handling Certificate Pinning in API Analysis.

The distribution ships three front ends over the same engine: mitmproxy (a terminal interface with a scrollable flow list), mitmweb (the same in a browser), and mitmdump (no interface, output to stdout or a file). Use mitmproxy while exploring and mitmdump once you know what you want.

Setting Up an Emulator You Control

Four steps to trust a capture certificate Run the proxy, point the emulator network settings at the host, download the certificate from the proxy's own page, and install it into the system trust store. A closing note limits this to devices you own. Setting up trust on an emulator you own1. Run proxymitmdump -p 8080on the host2. Route itWi-Fi proxy set tothe host IP3. Get the CAbrowse to mitm.itsave the PEM file4. Trust itcopy into thesystem storeOnly on an emulator image or debug device you controlnever install a capture root on a shared, borrowed or production handset
Four ordered steps turn a blank emulator into a capture target: start the proxy, route the traffic, fetch the certificate, then place it where the platform looks for roots.

Install the proxy and run it once so it generates its certificate authority into ~/.mitmproxy/:

python -m pip install "mitmproxy>=10.2"
mitmdump --listen-port 8080 --set confdir=~/.mitmproxy

Boot the emulator with the proxy set at the command line and the system partition writable. The Wi-Fi proxy field in the settings UI works for some traffic and is silently ignored by others, so the flag is more reliable:

emulator -avd Pixel_7_API_34 -http-proxy http://10.0.2.2:8080 -writable-system

10.0.2.2 is the fixed alias for the host machine as seen from inside an Android emulator. Use a Google APIs system image, not a Google Play one — Play images refuse to remount the system partition, which blocks the certificate step entirely.

Android 7 and later ignores user-installed certificates for app traffic unless the app opts in, so the certificate has to go into the system store. Android looks up roots by a filename derived from the subject hash:

HASH=$(openssl x509 -inform PEM -subject_hash_old -in ~/.mitmproxy/mitmproxy-ca-cert.pem | head -1)
cp ~/.mitmproxy/mitmproxy-ca-cert.pem "/tmp/${HASH}.0"

adb root
adb remount
adb push "/tmp/${HASH}.0" /system/etc/security/cacerts/
adb shell chmod 644 "/system/etc/security/cacerts/${HASH}.0"
adb reboot

Note -subject_hash_old: Android expects the pre-OpenSSL-1.0 hash format, and using the modern one produces a file the platform never looks at. After the reboot, the certificate should appear under system credentials in the device settings.

If you would rather not touch the system partition, the alternative is a desktop build of the same app. Desktop clients honour the OS proxy settings and the OS trust store, which on macOS and Linux means one security add-trusted-cert or one file in /usr/local/share/ca-certificates/, and no emulator at all.

Filtering to One Host and Reading a Flow

An idle phone generates a startling amount of background noise — telemetry, push registration, ad networks, crash reporters. Filter before you read.

mitmproxy --listen-port 8080 --set confdir=~/.mitmproxy "~d api.example.com"

The filter language is small and worth knowing four expressions of:

  • ~d example.com — domain matches
  • ~u /v3/catalog — URL contains
  • ~m POST — method
  • ~t json — response Content-Type contains "json"

They combine with &, | and !, so "~d api.example.com & ~t json & !~u /telemetry" is a normal working filter. In the interactive interface, f sets the filter live, arrow keys move through the list, Enter opens a flow, Tab switches between request and response, and q backs out.

An opened flow shows four things worth recording: the full URL including its query string, the request headers, the request body if there is one, and the response body. What you are looking for is the pattern rather than the values — which segment of the path is the version, which header carries the token, which query parameter is the page cursor.

Reading a flow list productively is mostly about ignoring things. A cold app start produces fifty to two hundred flows, and perhaps six of them matter. Three habits cut that down quickly. Clear the list immediately before performing the action you care about, so everything remaining is causally related to it. Sort or scan by response size — the screen you just opened is usually the largest JSON body in the window. And read the request that immediately precedes the interesting one, because auth and configuration calls cluster right before the first data call and are the ones you will need to reproduce first.

Two export commands turn a flow into something you can replay. From an open flow, e exports; the curl option produces a command you can paste into a terminal, and the raw option gives you the exact bytes. For a capture you intend to script against, save the whole session instead:

mitmdump --listen-port 8080 --set confdir=~/.mitmproxy \
  "~d api.example.com & ~t json" -w catalog-session.mitm

.mitm files replay through mitmdump -r catalog-session.mitm, which is useful for re-reading a capture without the app running.

An Addon That Logs Matching Requests to JSONL

The interface is for exploring. For anything repeatable, write an addon — a Python module mitmproxy imports, exposing hook functions it calls as flows complete. This one writes one JSON object per line for every response from a target host, which is the format the rest of the pipeline can read directly.

Addon filtering before the capture file The proxy core hands each completed flow to a response hook. A host filter splits the stream: matching flows are appended as JSON lines, everything else is dropped without being written. Proxy coreevery flowResponse hookdef response(flow)Host filterflow.request.hostends with targetAppend linecapture.jsonlDropnothing writtenmatchother
Every flow passes through the response hook, but only the ones whose host matches the filter are serialised — everything else is discarded before it touches disk.
"""mitmproxy addon: append matching flows to a JSONL capture file.

Run with:  mitmdump -s capture_addon.py --listen-port 8080
"""
from __future__ import annotations

import json
from datetime import datetime, timezone
from pathlib import Path

from mitmproxy import http

TARGET_SUFFIX = "api.example.com"
OUTPUT_PATH = Path("capture.jsonl")
MAX_BODY_BYTES = 200_000

SKIP_HEADERS = {"content-length", "connection", "accept-encoding", "host"}


def _headers(items: list[tuple[str, str]]) -> dict[str, str]:
    return {k: v for k, v in items if k.lower() not in SKIP_HEADERS}


def _body(raw: bytes) -> str | None:
    if not raw:
        return None
    if len(raw) > MAX_BODY_BYTES:
        return f"<truncated {len(raw)} bytes>"
    try:
        return raw.decode("utf-8")
    except UnicodeDecodeError:
        return "<binary>"


def response(flow: http.HTTPFlow) -> None:
    """Called by mitmproxy once a response has been fully received."""
    if not flow.request.pretty_host.endswith(TARGET_SUFFIX):
        return

    record = {
        "captured_at": datetime.now(timezone.utc).isoformat(),
        "method": flow.request.method,
        "url": flow.request.pretty_url,
        "path": flow.request.path,
        "request_headers": _headers(list(flow.request.headers.items())),
        "request_body": _body(flow.request.raw_content or b""),
        "status": flow.response.status_code if flow.response else None,
        "response_headers": _headers(
            list(flow.response.headers.items()) if flow.response else []
        ),
        "response_body": _body(flow.response.raw_content or b"") if flow.response else None,
    }

    with OUTPUT_PATH.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, ensure_ascii=False) + "\n")

Three details make this usable rather than merely working. pretty_host resolves the hostname from the SNI or Host header rather than the raw IP, so the suffix check matches what you expect. raw_content returns the decoded body — content can raise on malformed compression, and raw_content will not. And SKIP_HEADERS drops the connection-level headers immediately, because those are the ones that break a replayed request if they survive into your client code.

Reading the file back is one line per record:

import json
from pathlib import Path


def load_capture(path: Path) -> list[dict]:
    with path.open(encoding="utf-8") as handle:
        return [json.loads(line) for line in handle if line.strip()]


if __name__ == "__main__":
    flows = load_capture(Path("capture.jsonl"))
    catalog = [f for f in flows if "/catalog/" in f["path"] and f["status"] == 200]
    print(f"{len(flows)} flows captured, {len(catalog)} catalog calls")
    for flow in catalog[:3]:
        print(flow["method"], flow["url"])

Turning one of those records into a client that runs on a schedule is covered in Replaying Mobile API Requests in Python.

Addons expose more hooks than response, and two of them are worth knowing. request(flow) runs before the request leaves for the upstream server, which is where you would rewrite a header to test whether the backend actually reads it. tls_failed_client(data) fires when a client-side handshake collapses, which is the single most useful signal when nothing is being captured: an addon that logs the SNI hostname of every failed handshake tells you exactly which host refused your certificate, rather than leaving you to infer it from an empty flow list. Loading several addons at once is just repeated -s flags, and hooks from different modules run in the order the flags appear.

Edge Cases and Caveats

  • A capture file holds live credentials. capture.jsonl contains bearer tokens and session cookies in plain text. Keep it out of version control, and delete it when the analysis is done.
  • -writable-system does not survive a cold boot with data wipe. Wiping the AVD removes the certificate along with everything else. Script the install so re-running it is cheap.
  • Some apps use their own DNS or QUIC. A client that speaks HTTP/3 over UDP bypasses an HTTP proxy entirely. If flows are missing while the app clearly works, try blocking UDP 443 at the host firewall so the client falls back to TCP.
  • Background noise can drown the target. Airplane mode plus Wi-Fi only, and a filter set from the start, cuts the flow list by an order of magnitude.
  • pretty_url can differ from what was on the wire. It reconstructs the absolute URL for readability. When exact bytes matter, read flow.request.path and the Host header separately.
  • Response bodies may be compressed. raw_content gives you decoded bytes, but a body flagged as JSON is not guaranteed to parse — always guard the decode, as the addon above does.
  • Large media inflate the file fast. The MAX_BODY_BYTES guard exists because a single image response can be larger than every JSON body in a session combined.

Frequently Asked Questions

Why does the app work fine but no flows appear? The traffic is not reaching the proxy. Confirm the emulator was started with the -http-proxy flag, that the port matches, and that nothing on the host is blocking the listener. If flows appear for other apps but not this one, the client is likely using its own network stack or HTTP/3.

What is the difference between mitmproxy, mitmweb and mitmdump? They share one engine and one filter language. mitmproxy is the terminal interface, mitmweb the browser interface, and mitmdump a non-interactive version for logging and addons. Anything you can do in one, you can do in the others.

Do I need to write an addon, or is a saved flow file enough? A .mitm file is enough for one-off inspection and replays cleanly through mitmdump -r. An addon is better when you want a stable, filtered, machine-readable record — JSONL is trivial to read from any script and does not depend on the mitmproxy version that wrote it.

Can I modify requests as they pass through? Yes — a request hook can rewrite headers, paths and bodies before they go upstream, which is how you test whether a specific header is actually required. Do it against your own account and your own session; treat it as a diagnostic, not a delivery mechanism.

Back to home