Scrapeless Wiki

What Is a Registrable Domain? – Definition and How It Works

Informational P3 what is a registrable domain

Learn what a registrable domain is, how the Public Suffix List defines it, and why string suffix logic gets it wrong.

A registrable domain β€” also called eTLD+1 β€” is a public suffix plus one label: the shortest name a person or organisation can actually register. It is the correct unit for "same site" decisions.

1. The Definition Depends on a List

The intuition "take the last two labels" fails immediately:

Host Registrable domain Why
www.example.com example.com Two labels correct here
www.example.co.uk example.co.uk co.uk is a public suffix
shop.example.com.au example.com.au Three labels
user.github.io user.github.io github.io is a public suffix
a.b.s3.amazonaws.com b.s3.amazonaws.com s3.amazonaws.com is a public suffix

The last two rows are the ones that break naive implementations. Private suffixes exist because separate customers control separate subdomains, and treating github.io as one site would let one user's pages act on another's.

2. The Public Suffix List

There is no algorithm that derives this. The set of public suffixes is a maintained list, covering both ICANN suffixes (registry TLDs and their second levels) and private ones (services delegating subdomains to independent parties).

Because it is a list, it changes. A library bundling a stale copy computes yesterday's answers β€” which is why the list version is worth pinning and updating deliberately rather than accidentally.

3. Doing It Correctly

import tldextract

for host in ["www.example.co.uk", "user.github.io", "a.b.s3.amazonaws.com"]:
    parts = tldextract.extract(host)
    print(host, "->", parts.registered_domain or host)

Use a library backed by the list. String manipulation on dots is wrong for a large and growing share of real hostnames.

4. Where It Matters

Cookie scope. A cookie may be set for a registrable domain but not for a public suffix β€” the rule that stops one site setting a cookie for all of .co.uk.

Crawl boundaries. "Stay on this site" means staying within the registrable domain. Comparing full hostnames treats www. and the bare domain as different sites; comparing the last two labels wanders across unrelated sites on shared suffixes.

Rate limiting. Politeness limits belong per registrable domain, because hundreds of subdomains usually share one set of servers. Limiting per hostname lets a crawler overwhelm a host it was trying to be polite to.

Deduplication and aggregation. Grouping results by site requires this unit, or blog.example.com and www.example.com are counted as separate publishers.

Same-origin reasoning. Related but distinct β€” an origin is scheme, host, and port exactly, with no suffix logic at all.

5. Common Mistakes

  • Splitting on dots and taking the last two labels. Wrong for every multi-label suffix.
  • Ignoring private suffixes, so unrelated tenants on a platform look like one site.
  • Bundling a stale list and never updating it.
  • Treating an IP address as a host with a suffix β€” it has none, and the whole address is the identity.
  • Missing the leading-dot and wildcard rules, including exception rules that override a wildcard.

6. Punycode and Case

Normalise before extracting: lowercase the host, and convert internationalised names to their ASCII form, since the list is expressed in that form. A host differing only in case or encoding must resolve to the same registrable domain, or per-domain state silently splits in two.

7. Summary

A registrable domain is a public suffix plus one label β€” the shortest independently registrable name β€” and it is defined by the maintained Public Suffix List, not by counting dots.

It is the right unit for cookie scope, crawl boundaries, rate limiting, and site-level grouping, so use a list-backed library, keep the list current, and normalise case and punycode before extracting.