Scrapeless Wiki

How Does a Proxy Server Work?

Informational P1 how does a proxy server work
Overlaps a live page. A published page already targets this keyword: https://www.scrapeless.com/en/blog/what-is-a-proxy-server. Publishing both risks cannibalisation — decide before this goes live.

Follow a request through a proxy step by step: configuration, CONNECT tunnels for HTTPS, authentication, pooling, and where latency comes from.

A proxy works by inserting itself between your client and the destination. The mechanics differ meaningfully depending on whether the traffic is plain HTTP or encrypted, and the difference explains most of what proxies can and cannot do.

1. The Basic Path

  1. Your client is configured to use the proxy.
  2. Instead of connecting to the destination, it connects to the proxy.
  3. The proxy makes the request to the destination from its own address.
  4. The destination responds to the proxy.
  5. The proxy relays the response back.

The destination's logs record the proxy's IP. It has no direct view of yours.

2. Plain HTTP

For unencrypted HTTP, the client sends the absolute URL in the request line rather than a path:

GET http://example.com/products HTTP/1.1
Host: example.com

The proxy reads this, opens its own connection to example.com, requests /products, and returns what it gets.

Because everything is in the clear, the proxy can cache the response, filter it, log the full URL, or rewrite headers. Old caching proxies existed for exactly this.

3. HTTPS and the CONNECT Tunnel

For HTTPS, the proxy cannot read anything — and the protocol accounts for that.

CONNECT example.com:443 HTTP/1.1

The proxy opens a TCP connection to example.com:443 and replies:

HTTP/1.1 200 Connection Established

From that point it copies bytes in both directions without interpreting them. The TLS handshake happens end to end between your client and the destination, straight through the tunnel.

Two consequences follow, and they are the important ones:

  • The proxy sees only the hostname, port, timing and byte volume. Not the path, not the headers, not the body.
  • The destination sees your client's genuine TLS fingerprint, because your client performed the handshake. A proxy does not disguise what software you are running — which is why rotating IPs does nothing for a scraper identified at the TLS layer.

4. Authentication

Proxies identify clients in one of two ways:

Credentials in the request:

Proxy-Authorization: Basic <base64 user:pass>

Most libraries accept them in the URL — http://user:[email protected]:8080 — and pass them along as the header.

IP allowlisting. The provider authorises your server's address, and no credentials travel with the request. Simpler, and unusable from a dynamic address.

Credentials frequently encode routing options too, which is how a single endpoint serves many configurations:

user-country-de-session-abc123:[email protected]:8080

Here the username selects a German exit and pins a session, so successive requests reuse the same address.

5. Rotating and Sticky Sessions

A rotating pool is one endpoint in front of many addresses.

  • Per-request rotation. Every request exits from a different IP. Right for independent fetches.
  • Sticky sessions. A session identifier holds one address for a period — minutes, typically. Required whenever state matters: logging in, paginating, or anything with a cart.

Getting this wrong is a common and confusing bug. A login that succeeds and a next page that returns "session expired" is usually one request arriving from a different country than the last.

6. Where the Latency Goes

A proxied request travels client → proxy → destination and back, so it is at minimum the sum of two legs:

  • Datacenter proxies add tens of milliseconds. The hop is between well-connected facilities.
  • Residential proxies add hundreds. The exit is a consumer connection with consumer upstream bandwidth and variable reliability.
  • Mobile proxies add more still, and vary the most.

Additional connections mean additional TLS handshakes if you do not reuse them, so connection pooling matters more through a proxy than without one. Per-request rotation is at odds with connection reuse — every new IP is a new connection and a new handshake, which is a real part of why rotation is slow.

7. Real-World Examples

  • A CONNECT tunnel through which a provider logged only example.com:443 — proof of what a proxy can actually observe.
  • A pipeline blocked identically across thousands of residential IPs, because the TLS fingerprint travelled through the tunnel unchanged.
  • A cart that emptied on every page until a sticky session pinned the exit address.
  • A crawler running at a third of its expected rate because per-request rotation prevented any connection reuse.
  • Credentials in a URL leaking into application logs, the standard way proxy passwords escape.

8. Summary

A proxy accepts your request, reissues it from its own address, and returns the response. Plain HTTP passes through as an absolute URL the proxy can read, cache and modify. HTTPS uses a CONNECT tunnel the proxy relays blindly, so it observes the hostname and the timing and nothing else.

Because the TLS handshake is end to end, the destination sees your client's real fingerprint no matter which IP it arrives from — a proxy changes where a request comes from, not what made it.

Authentication is by credentials or by allowlisted IP, and credentials commonly carry routing options such as country and session pinning. Choose per-request rotation for independent fetches and sticky sessions for anything stateful, and expect proxied latency to be additive, with residential exits an order of magnitude slower than datacenter ones.