Scrapeless Wiki

What Is an HTTP Header? – Definition and How It Works

Informational P1 what is an http header

Learn what HTTP headers are, the request and response headers that matter most, how ordering reveals a client, and where headers commonly go wrong.

An HTTP header is a name-value pair carrying metadata about a request or response β€” who is asking, what format is expected, how the result may be cached, and how the body should be interpreted.

1. What Is an HTTP Header?

Headers describe the message; the body is the message.

  • Key idea: separate metadata from payload so intermediaries can act on it without parsing the body.
  • Mechanism: name-value pairs sent before the body, case-insensitive by name.
  • Goal: negotiate format, carry credentials, control caching, and describe content.

Example

GET /api/v1/items HTTP/1.1
Host: api.example.com
Accept: application/json
Accept-Language: en-US,en;q=0.9
Authorization: Bearer <token>
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...

2. Request Headers That Matter

Header Purpose
Host Which site on a shared address is being addressed
Accept Formats the client can handle
Accept-Language Preferred languages, which often changes the response
Authorization Credentials, commonly a bearer token
Content-Type Format of the body being sent
User-Agent Client identity, self-declared
Referer The page the request came from
Cookie Stored state from previous responses

3. Response Headers That Matter

  • Content-Type β€” how to interpret the body, including character encoding.
  • Set-Cookie β€” state the client should return on later requests.
  • Cache-Control and ETag β€” whether and how the response may be reused.
  • Location β€” the target of a redirect.
  • Retry-After β€” when a rate-limited or unavailable resource may be requested again.

4. Headers Are Also a Fingerprint

Real browsers send a consistent set of headers, in a characteristic order, with plausible values. HTTP client libraries send fewer, in a different order, often omitting ones no browser would leave out.

That difference is measurable, which is why a request claiming to be a browser in its user agent while sending a library's header profile is self-contradictory. Consistency across the whole set matters more than any single header's value.

5. Where Headers Go Wrong

  • Wrong content type. Sending JSON without Content-Type: application/json makes many servers reject or misparse the body.
  • Missing Accept. Some APIs return HTML instead of JSON when the client does not state a preference.
  • Case assumptions. Header names are case-insensitive; code that looks for an exact case can miss them.
  • Duplicate headers. Some may legitimately repeat, others must not; behaviour varies by server.
  • Secrets in the wrong place. Credentials belong in headers, not in query strings, which are logged by proxies and stored in browser history.

6. Custom Headers

Applications define their own, historically with an X- prefix, now usually without. A custom header carrying an API key β€” for example a token header specific to a service β€” is common, and the header name is part of that API's contract: sending the right key under the wrong name authenticates nothing.

7. Common Misconceptions

  • "Headers are optional metadata." Several change the response entirely, including language, format, and authentication.
  • "User agent identifies the client." It is self-declared and freely set; it is a claim, not evidence.
  • "Header order is irrelevant." Semantically yes; as a fingerprint, order is one of the more revealing signals.
  • "Bigger headers are harmless." Servers enforce size limits, and oversized header blocks are rejected outright.

8. Summary

HTTP headers carry the metadata that decides how a message is authenticated, interpreted, cached, and routed. A handful β€” host, accept, authorization, content type, cookie β€” cover most day-to-day work.

Beyond their stated meaning, the set a client sends is characteristic of that client, so internal consistency matters as much as any individual value.