Python Web Scraping
Reading layout

Handling Certificate Pinning in API Analysis

Certificate pinning is the reason a capture session set up correctly, as described in Scraping Mobile App APIs, can still produce an empty flow list.

Where a pinned handshake stops Four stages run left to right: the client hello, the certificate the proxy presents, the pin comparison against a compiled key list, and the abort. A closing note explains what the proxy log looks like afterwards. A pinned handshake, stage by stageClientHelloapp opens TLSto the API hostCertificateproxy presents itsown leaf keyPin checkSHA-256 of the keyvs compiled listNo matchhandshake abortedsocket closedThe proxy shows a connection with no request inside itno method, no path, no headers — the client gave up before sending any of them
A pinned client compares the presented public key against a list compiled into the app and closes the socket on mismatch — the HTTP request is never sent, so there is nothing to record.

A pinned client does not ask the operating system whether a certificate is acceptable. It carries its own short list of public keys, compares what the server presented against that list, and closes the socket on a mismatch. Your proxy's certificate is not on the list and never will be, so the handshake dies before any HTTP request is written — which is why the log shows a connection and nothing inside it. The useful work at that point is diagnosis: confirm it really is a pin, then take one of the documented routes rather than trying to remove it.

To be explicit about scope: modifying, repackaging or instrumenting somebody else's production app to strip its pins is out of scope on this page. It is a control the vendor deliberately shipped, defeating it commonly breaches the terms you accepted on install, and in several jurisdictions it touches computer-misuse law. This page covers how to recognise a pin and what the honest options are.

How Pinning Changes the Handshake

An ordinary TLS client validates a chain: the leaf certificate is signed by an intermediate, which is signed by a root, and the root must be present in the platform trust store. Adding your own root to a device you own is what makes interception work at all — you have extended the set of chains the client considers valid.

Pinning inserts a second, narrower check that the trust store cannot influence. After chain validation passes, the client hashes the certificate's subject public key info — the SPKI block — and compares that digest against a set compiled into the app. On Android this is usually a network_security_config.xml entry or an OkHttp CertificatePinner; on iOS it is typically a delegate method that inspects the server trust object. Either way the comparison happens inside the app, against data inside the app.

That placement explains the symptom precisely. The comparison runs during the handshake, before the client has sent a request line, so there is no method, no path and no headers to record. A proxy that logs everything still logs nothing, because nothing was said.

It is also worth being precise about what gets pinned, because the terminology causes confusion. Pinning the leaf certificate breaks every time the vendor renews, so almost nobody does it. Pinning the SPKI hash is the standard approach: the key survives certificate renewal, so the pin outlives routine rotation. A third variant pins the intermediate or the issuing CA, which is looser — it accepts any certificate that authority issues, and it still rejects a locally generated root.

Three properties of real deployments are worth knowing:

  • Pins are per host, not per app. A vendor typically pins the API domain and leaves the CDN, the analytics endpoint and the image host unpinned. Partial visibility is the normal picture.
  • Pins expire. They are tied to key rotation, so apps ship pin sets with backup entries and a validity window. An old build can fail against the live service for exactly this reason.
  • Pinning is not encryption. It changes who the client will talk to, not what is readable. There is no key to find.

Recognising the Symptom

Symptoms and what they indicate Three rows pair an observed symptom on the left with the condition it points to on the right: handshake errors, instant connection failure in the app, and some screens working while others fail. What you seeWhat it points toHandshake error in the loga TLS failure line, never a method or pathApp says no connection at onceno spinner, no timeout, no retrySome screens load, others failthe login works but the feed stays emptyUntrusted root, or a pincheck the trust store first, it is commonerThe client closed the socketnothing left the device to be inspectedOnly some hosts are pinnedpins are set per domain, not per app
Read the symptom before reaching for a conclusion: an untrusted root and a pinned host fail differently, and partial failure across screens points at a per-host pin set.

Before concluding anything, rule out the far more common cause: a certificate authority that was never installed correctly. The two failures look similar in a screenshot and completely different in a log.

mitmdump --listen-port 8080 --set confdir=~/.mitmproxy --set flow_detail=3 2>&1 \
  | grep -iE "client(dis)?connect|tls|handshake|error"

Read the output against this table:

ObservationUntrusted rootPinned host
Which hosts failevery hostone host, others fine
Timingfails on first connectionfails after chain validation
App behaviournothing loads at allsome screens load, one does not
Browser on the same devicealso failsloads normally
After reinstalling the CAfixedunchanged

The browser row is the fastest test. Open the same device's browser to any HTTPS page while the proxy runs. If the browser is intercepted cleanly and the app's API host is not, your trust store is fine and the app is applying its own rule.

You can also confirm from the server side that the certificate the API normally presents is ordinary, which distinguishes "the app rejected my certificate" from "the server did something unusual":

openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -subject -dates

A public issuer and a normal validity window mean the server is behaving conventionally and the restriction lives in the client. That is a pin.

One more check separates a pin from a client that simply refuses proxies for other reasons. Point the emulator at the proxy but add a passthrough rule for the API host, so the proxy tunnels those bytes without terminating TLS:

mitmdump --listen-port 8080 --set confdir=~/.mitmproxy \
  --ignore-hosts '^api\.example\.com:443$'

If the app works normally with that rule and fails without it, the client is rejecting your certificate specifically — the connection path, DNS and network route are all fine. If it fails either way, the problem is somewhere else entirely: a captive network, an app that refuses to run when a proxy is configured at all, or a client using a transport the proxy never sees. Passthrough gives you no readable data, but it is the cheapest way to isolate the variable, and it costs one flag.

The Documented-API Path

Options after finding a pinned host Four stacked rows in priority order: use the vendor's documented API, ask the vendor for access, analyse a build you own, and a final row marking pin removal on other people's apps as out of scope. Where to go once a host turns out to be pinned1234Use the vendor's documented APIkeys, quotas and terms you can rely on for yearsAsk the vendor for accessmany publish a partner, research or bulk export routeAnalyse a build and device you ownyour own copy, your own emulator image, your own CAOut of scope: defeating someone else's pinsnot covered here — stop, and take one of the routes above
Ranked from cheapest and most durable to worst: the documented API beats a self-owned debug build, which beats guessing — and stripping pins from someone else's app is not on the list.

A pinned host is a signal, and the productive response is to look for the interface that was meant for programmatic access. It exists more often than people assume, and it is strictly better than a capture: it is documented, it has a support channel, and it does not break when the app ships a new build.

Look in four places, in order of how often they pay off. The developer subdomain (developers.example.com, api.example.com/docs) is the obvious one. Machine-readable descriptions are the next: many services publish an OpenAPI document at a predictable path, and finding one gives you the entire endpoint list without a single capture. Third, partner and research programmes — plenty of companies grant bulk or historical access on request that no amount of interception would give you. Fourth, bulk exports: data dumps, sitemaps and feeds that sidestep per-record fetching entirely.

import httpx

DOCS_PATHS = (
    "/openapi.json",
    "/swagger.json",
    "/v3/api-docs",
    "/.well-known/openapi.json",
)


def find_api_description(base_url: str) -> tuple[str, int] | None:
    """Probe a small set of conventional paths for a machine-readable API spec."""
    headers = {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "Accept": "application/json",
    }
    with httpx.Client(timeout=10.0, headers=headers, follow_redirects=True) as client:
        for path in DOCS_PATHS:
            url = base_url.rstrip("/") + path
            try:
                response = client.get(url)
            except httpx.HTTPError:
                continue
            content_type = response.headers.get("content-type", "")
            if response.status_code == 200 and "json" in content_type:
                return url, len(response.content)
    return None


if __name__ == "__main__":
    found = find_api_description("https://httpbin.org")
    print(found if found else "no machine-readable description at the usual paths")

When a description does turn up, read the rate limits and the authentication section before anything else. A documented endpoint with a published quota removes the entire class of problems that makes captured traffic fragile: you know how fast you may go, you know how long a key lasts, and a breaking change arrives with a version number and usually a deprecation notice rather than as a column of nulls. It is common for the documented interface to be a superset of what the app calls, with filtering, sorting and bulk parameters the mobile client never uses — which frequently means fewer requests for the same dataset than interception would have given you.

Four polite probes is a reasonable amount of curiosity; a hundred is a scan. If nothing turns up, write to the vendor. A short message describing what data you need, at what volume and for what purpose is answered surprisingly often, and an approved key beats a fragile capture in every dimension that matters over a year.

Working Within a Build You Control

There is one legitimate technical route left, and it is narrow: analyse software you own, on hardware you own. If the vendor publishes a debug build, or you are analysing your own organisation's app, or the app is open source and you can compile it yourself, then the pin set is yours to configure. A debug build commonly ships an Android network_security_config.xml that already trusts user-added roots, and a source build lets you point the pin at your own CA before compiling.

This is not a workaround for the general case. It applies when you already have the right to modify the client, and it produces the same capture the vendor's own engineers get. If none of those conditions hold, the option is not available and the correct move is one of the routes above — or stopping.

Two related paths are sometimes mistaken for this one and are not the same thing. Running an older version of the app is not a legitimate route around a pin: an old build ships old pins, so it fails against the current service rather than accepting your certificate, and hunting archived binaries for one that happens to predate the pin is exactly the boundary this page draws. Similarly, a vendor-provided staging or sandbox environment is a genuine option — many are deliberately unpinned and populated with test data — but its data is synthetic, so it is useful for learning the request shape and useless as a source of records.

Knowing when to stop is part of the skill. If the host is pinned, no documented interface exists, the vendor does not answer, and you do not control the build, then the exercise is over. That is not a failure of technique; it is the control working exactly as intended, and the time is better spent on a source that wants to be read.

Edge Cases and Caveats

  • Partial success is the norm. Login and images work, the catalogue endpoint does not. That pattern is a per-host pin, not a broken setup, and it means part of the API may still be readable.
  • A wrong hash format looks identical to a pin. Android matches system roots by the old-style subject hash. Using -subject_hash instead of -subject_hash_old produces a file the platform ignores, and every host then fails — which is an untrusted root, not a pin.
  • Old builds fail against live services. Pin sets have expiry dates. An app pulled from an archive may fail its handshake even with no proxy present.
  • Pinning does not imply TLS fingerprinting. They are separate controls. A host can pin and still accept any client TLS profile once you use a real certificate — the fingerprint layer is described in TLS and JA3 Fingerprint Evasion.
  • The web version may not be pinned. Browsers have effectively abandoned per-site pinning, so the same data behind the site's own JSON layer is often reachable with no device involved at all.
  • Do not leave a capture root installed. Whatever the outcome, remove the CA from any device that will be used for anything else afterwards.

Frequently Asked Questions

How do I tell pinning apart from a certificate that was not installed properly? Test a second host. An untrusted root breaks every HTTPS connection on the device, including the browser; a pin breaks exactly the hosts that carry one while everything else is captured normally. If reinstalling the CA fixes nothing and other traffic is visible, it is a pin.

Is it illegal to bypass certificate pinning? That depends on jurisdiction, on the terms you accepted, and on whose app and device are involved, and this is not legal advice. What is clear is that a pin is an access control the vendor deliberately shipped, so treat removing it from somebody else's production app as out of bounds and use the documented interface instead.

Can I just use the website instead? Frequently, yes. Browser vendors have deprecated public key pinning for websites, so the same backend reached through the site's own JSON layer is usually interceptable and often serves identical records. Check that path before investing anything further in the app.

Will the app tell me it is pinned? Not usually. Most clients surface a generic connection error, because a specific message would be a hint to an attacker. The proxy log is the honest source: a connection opened and closed with no request inside it is the signature.

Back to home