Scrapeless Wiki

What Is a User Agent? – Definition and How It Works

Informational P1 what is a user agent
Overlaps a live page. A published page already targets this keyword: https://www.scrapeless.com/en/blog/user-agent. Publishing both risks cannibalisation — decide before this goes live.

Learn what a user agent string is, how to read its historical baggage, what Client Hints replace it with, and why rotating it rarely helps.

The User-Agent header is a self-declared description of the client making a request. It is the oldest identification mechanism on the web, and — because it is self-declared — one of the weakest.

1. What Is a User Agent?

It is a string sent with every HTTP request describing the client software.

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
  • Key idea: a claim about the client, trivially editable by the client.
  • Sent by: browsers, HTTP libraries, crawlers, apps — anything making requests.
  • Used for: analytics, feature decisions, bot identification, content negotiation.

2. Why It Looks Absurd

The string above claims to be Mozilla, AppleWebKit, KHTML, Gecko, Chrome and Safari simultaneously. That is not an error; it is thirty years of compatibility sediment.

Every browser added the previous market leader's identifiers so that servers doing crude string matching would not serve them a degraded page. Nobody could ever remove a token, because some server somewhere still checks for it.

The practical lesson: parse for what you need, never match the whole string. And treat everything before the meaningful part as noise.

3. Reading One

Token Means
Mozilla/5.0 Historical, present in essentially all of them
(Windows NT 10.0; Win64; x64) Platform
AppleWebKit/537.36 (KHTML, like Gecko) Engine lineage, mostly historical
Chrome/120.0.0.0 The actual browser and version
Safari/537.36 Historical compatibility

Bot user agents are usually far more honest, because identifying themselves is the point:

Googlebot/2.1 (+http://www.google.com/bot.html)
GPTBot/1.0 (+https://openai.com/gptbot)
python-requests/2.31.0

Note the URL — well-behaved crawlers include a page explaining themselves. That is a convention worth following if you operate one.

4. Client Hints

Browsers are moving away from the string. Chrome now freezes most of the User-Agent's detail and offers Sec-CH-UA-* headers instead:

Sec-CH-UA: "Chromium";v="120", "Google Chrome";v="120"
Sec-CH-UA-Mobile: ?0
Sec-CH-UA-Platform: "Windows"

The design difference is that hints are requested. A server asks for the high-entropy values it needs via Accept-CH, rather than every client broadcasting a detailed profile to every site. Less passive fingerprinting surface, and cleaner parsing.

The User-Agent header still exists, and will for a long time. It just carries progressively less information.

5. Why Rotating It Rarely Helps

A persistent belief holds that rotating user agents defeats detection. It does not, for two reasons.

It is one signal among dozens. Detection weighs TLS fingerprint, HTTP/2 frame ordering, header order, browser attributes and behaviour. The user agent is the easiest of these to change, which is exactly why it is trusted least.

It creates contradictions. Setting a Chrome user agent on a Python request produces a claim that contradicts the TLS handshake — and the handshake happened first, before the header was even sent. A request claiming Chrome 120 with requests' fingerprint is not ambiguous; it is impossible, and impossible is a far stronger signal than unusual.

Random rotation compounds this: a single session presenting five different browsers is more anomalous than one presenting a consistent, honest client.

If you set one, set it to something coherent with everything else you send — and if you cannot make it coherent, an honest default is often the safer choice.

6. Legitimate Uses

  • Analytics. Approximate browser and platform breakdowns.
  • Crawler identification. Recognising Googlebot, GPTBot and others, usually verified by reverse DNS rather than trusted outright.
  • Compatibility shims. Rarely necessary now, but real for known engine bugs.
  • Identifying your own bot. If you run a crawler, say so and link to a page explaining it. It is the difference between a courteous crawler and an anonymous one.

7. Real-World Examples

  • A "Chrome" request with a Python TLS fingerprint — the contradiction, not the header, is what flagged it.
  • A site serving a legacy layout because of a decades-old substring check.
  • A crawler blocked for anonymity and unblocked after identifying itself honestly with a contact URL.
  • Rotation making detection easier, when one session cycled through incompatible platforms.
  • A parser broken by Chrome's UA reduction, having relied on a minor version that no longer varies.

8. Summary

The User-Agent header is a self-reported description of the client, shaped by decades of compatibility workarounds into a string that claims several browsers at once. Parse the token you need; never match the whole thing.

Client Hints are replacing it with requested, structured headers, reducing what every client broadcasts by default.

Because it is self-declared and trivially editable, it is a weak signal on its own — and rotating it typically hurts rather than helps, because a claim that contradicts the TLS handshake, the header order or the browser environment is more revealing than any honest value would have been.