How to Identify Which Anti-Bot a Site Uses

Informational P2 how to identify which anti-bot a site uses

Learn the six signal classes that reveal which anti-bot system protects a site, and how to score them by confidence.

Identifying the protection in front of a site is a fingerprinting exercise in reverse: the defence leaves observable traces in cookies, headers, URLs, page content, the DOM, and the JavaScript environment.

1. Why Identify It At All

Knowing the vendor tells you what to expect before you spend any effort:

Whether the obstacle is a static rule or an active browser challenge.
Whether a plain HTTP client can ever work, or a real browser is required.
Whether the challenge is solvable at all, or the site simply should not be collected.
What the failure signatures will look like, so a block is not mistaken for a layout change.

It also prevents the most common waste: building an elaborate browser pipeline for a site whose only protection was a rate limit.

2. The Six Signal Classes

Class	Where to look	Strength
Cookie	Set-Cookie names	Very high — usually vendor-unique
URL	Script and endpoint paths	High
Window	JavaScript globals the script defines	High
Header	Response headers	Medium to high
Content	Strings in the HTML	Low — easily incidental
DOM	Injected elements, iframes, attributes	Medium

Cookie names are the strongest single signal because vendors need a stable, namespaced name and cannot easily randomise it. Content matches are the weakest — a page mentioning a vendor's name might merely link to a blog post about it.

3. Score Confidence, Don't Assert

A robust identification assigns each signal a confidence and combines them, rather than treating any single match as proof:

SIGNALS = {
    "aws_waf": [
        ("cookie", "aws-waf-token", 100),
        ("cookie", "awswaf",        100),
        ("url",    "/challenge.js",  100),
        ("url",    "/telemetry",      90),
        ("content","awswaf",          50),
    ],
    "threatmetrix": [
        ("url",    "fp/check.js",     95),
        ("window", "BNQL.Config",     95),
        ("url",    "org_id=",         85),
        ("content","ThreatMetrix",     50),
    ],
}

def identify(observed):           # observed: {"cookie": {...}, "url": [...], ...}
    out = {}
    for vendor, sigs in SIGNALS.items():
        hits = [(k, needle, c) for k, needle, c in sigs if _matches(observed, k, needle)]
        if hits:
            out[vendor] = {"confidence": max(c for _, _, c in hits), "signals": hits}
    return dict(sorted(out.items(), key=lambda kv: -kv[1]["confidence"]))

Taking the maximum confidence rather than summing avoids the trap where five weak content matches outweigh one definitive cookie.

4. Capturing the Observations

A plain HTTP request sees cookies, headers, URLs in the markup, and content. It cannot see window globals or dynamically injected DOM, because nothing executed.

A browser session sees everything, and a HAR export captures the request side of it:

context = browser.new_context(record_har_path="probe.har", record_har_content="omit")
page = context.new_page()
page.goto(url, wait_until="networkidle")
globals_found = page.evaluate("""() => {
  const probes = ['BNQL','_pxAppId','__CF$cv$params','kpsdk','dd_cookie_test'];
  return probes.filter(p => p.split('.').reduce((o,k)=>o&&o[k], window) !== undefined);
}""")
cookies = [c["name"] for c in context.cookies()]
context.close()

Two requests are worth making in every probe: one plain HTTP and one full browser. The difference between them is itself diagnostic — content that only appears in the browser means client-side rendering or a challenge that must execute.

5. Read the Difficulty, Not Just the Name

Vendors differ enormously in what they require, and that matters more than the label:

Rule and reputation based — a WAF blocking by IP, header, or rate. Often passes with ordinary polite requests from a decent address.
Passive fingerprinting — collects signals and scores silently. No visible challenge; you simply get different or empty content.
Active challenge — runs JavaScript that must complete before content is served. A plain HTTP client cannot pass.
Interactive CAPTCHA — requires a human-style interaction.
Device and fraud identification — services like ThreatMetrix, aimed at fraud scoring rather than blocking, which often do not block at all.

The last category is frequently misread as a blocker when it is only observing.

6. Confirm With Failure Signatures

An identification is a hypothesis until the failure mode matches. Check that:

The status code and body match the vendor's known challenge response.
The block appears at the expected point — first request, or after N requests.
A browser session succeeds where a plain client fails, if an active challenge is indicated.
A different egress address changes the outcome, if reputation is indicated.

If the observed behaviour disagrees with the identification, trust the behaviour.

7. What Not to Conclude

Presence of a vendor does not mean you are blocked — many sites run bot management in monitoring mode. Absence of signals does not mean no protection, since server-side scoring leaves no client trace. And multiple vendors on one site is normal: a CDN WAF plus a specialist bot product plus a fraud service is a common stack.

8. Summary

Identify a site's anti-bot by collecting six classes of signal — cookies, URLs, window globals, headers, content strings, and DOM markers — and scoring them by confidence rather than trusting any single match.

Probe with both a plain HTTP client and a real browser, because the difference between them reveals active challenges; read the vendor's category rather than its name to know whether you face a rule, a passive score, or a challenge; and treat the identification as a hypothesis until the observed failure signature agrees.