How to Set a User Agent – Practice and Pitfalls
Learn how to set a user agent in common clients, why honest identification beats imitation, and how a mismatched string makes things worse.
The user agent is a header describing the client. Setting it is trivial; setting it well is a decision about whether to identify honestly or imitate something you are not.
1. How to Set It
# requests
requests.get(url, headers={"User-Agent": "example-collector/1.0 (+https://example.com/bot)"})
// fetch in Node
await fetch(url, { headers: { 'user-agent': 'example-collector/1.0 (+https://example.com/bot)' } });
curl -A "example-collector/1.0 (+https://example.com/bot)" https://example.com/
Most clients also allow a default on a session or client object, which is preferable to repeating it per request.
2. Why the Default Is a Problem
Library defaults announce the library and its version. That is honest and, on some destinations, enough to attract different handling.
The instinct is to replace it with a browser string. That is where the trouble starts.
3. Why Imitation Backfires
A user agent is a claim. The connection also carries evidence: the TLS handshake, HTTP/2 framing, and header composition all belong to the software actually making the request.
Claiming to be a mainstream browser over a scripting library's handshake is a measurable contradiction — and contradiction is exactly what classification systems weigh most heavily. The imitation makes the client more conspicuous, not less.
4. Honest Identification Is the Better Default
A descriptive user agent naming your client and a contact URL lets an operator identify your traffic, allow it, or contact you rather than only block it:
example-collector/1.0 (+https://example.com/bot; [email protected])
For anything sustained this is more durable than any string chosen to blend in, because it makes an allow-list entry possible.
5. When a Browser String Is Correct
When you are actually using a browser. A real browser sends its own user agent, and it should not be overridden — doing so creates the same contradiction from the other direction.
Where the whole client genuinely matches a browser at every layer, the matching string is consistent rather than deceptive.
6. Setting It Consistently
If a user agent claims a platform, other headers should agree — accept-language plausible for that locale, client hint headers consistent, and header ordering characteristic of that client.
Partial imitation is more identifiable than none, which is why a coherent honest client beats a half-built impersonation.
7. What It Does Not Do
- It does not change the TLS or HTTP/2 fingerprint.
- It does not affect address reputation.
- It does not alter behaviour or pacing.
- It does not grant permission —
robots.txtand terms still govern.
8. Summary
Set the user agent through a client or session default, and prefer honest identification with a contact route over imitating a browser.
Imitation contradicts the evidence a connection already carries, which increases rather than reduces scrutiny — and a documented client is what makes an allow-list entry possible for collection that continues.