HTTP 410 Gone Explained – Causes and Fixes

Learn what HTTP 410 means, how it differs from 404, and why it is the one status that should stop a crawler retrying.

HTTP 410 Gone means the resource existed, has been deliberately removed, and is not coming back. It is a permanent, intentional statement — unlike 404, which merely says nothing is here now.

1. 410 Versus 404

The asymmetry is worth appreciating: a 404 is ambiguous, while a 410 is a promise. It is rare precisely because it requires the site to make that promise deliberately.

2. Why It Is Rare

Returning 410 takes work. A server must remember that a URL once existed and was removed, which means keeping tombstone records rather than simply not routing the path.

Most systems do not bother, so removed content usually returns 404. When you do receive a 410, take it seriously — someone configured it on purpose.

3. What It Means for a Crawler

410 is the clearest possible instruction, and acting on it is pure efficiency:

1. Remove the URL from the frontier permanently. Not "retry in a week" — never.
2. Mark the stored record as gone, with the date, rather than deleting the history.
3. Do not re-discover it. A link elsewhere on the site will re-add it unless you keep a permanent exclusion set.
4. Do not count it as a failure. It is a successful, informative response, and mixing it into an error rate makes a healthy crawl look broken.

Point 3 is the one most implementations get wrong, and it causes the same URL to be refetched indefinitely.

4. Recording It Correctly

TERMINAL_STATUSES = {410}          # never retry
SOFT_STATUSES = {404}              # retry with decay, then retire

def on_response(url, status, store):
    if status in TERMINAL_STATUSES:
        store.mark_gone(url, permanent=True)
        store.exclude_forever(url)
        return "gone"
    if status in SOFT_STATUSES:
        store.increment_miss(url)
        return "missing"
    return "ok"

Keeping the record rather than deleting it preserves the fact that the resource once existed — which is exactly the information a later analysis of removals needs.

5. Where It Appears Legitimately

• Expired listings — a job posting, auction, or classified past its life.
• Deleted user content removed on request.
• Retired API endpoints, where 410 is a considerate deprecation signal.
• Content removed for legal reasons, though 451 is the specific status for that.
• Deliberate SEO cleanup, where 410 asks search engines to deindex faster than 404.

6. When 410 Is Not Literal

Occasionally a site returns 410 as a blanket response to unwanted automated traffic, in which case it is not about the resource at all. The tells:

• Every URL returns 410, including the homepage.
• A browser loads the same URLs normally.
• A different egress IP succeeds.
• It began abruptly across the whole site.

That is a block wearing a 410, and it should be handled as a block. Any implementation that permanently excludes URLs on 410 needs this check, or a temporary block permanently deletes an entire site from the frontier — a genuinely destructive outcome from an otherwise correct rule.

7. Fix Checklist

1. Verify the URL is genuinely gone by loading it in a browser.
2. Check whether the whole site returns 410, which indicates a block rather than removal.
3. Try a different egress IP to distinguish the two.
4. If genuine, exclude the URL permanently and mark the record gone with a date.
5. Keep 410 out of your error rate.
6. Follow the site's own listings or sitemap for the replacement, if any.

8. Summary

HTTP 410 means a resource was deliberately and permanently removed — a promise, where 404 is only an observation.

Treat it as terminal: exclude the URL permanently, mark the stored record as gone rather than deleting it, and keep it out of error metrics — but first confirm the whole site is not returning 410, because a site-wide 410 is a block, and excluding on it would delete the entire site from your frontier.

Potential Related Glossary

• Web Crawling
• IP Rotation
• Residential Proxies
• Parsing