What Is Log File Analysis? – Definition and How It Works

Learn what server logs reveal about crawler behaviour, how to verify bot identity, and what to measure.

Log file analysis examines a web server's access logs to see what actually happened — which URLs were requested, by whom, and with what result. It is the only source of truth about crawler behaviour on a site.

1. Why Logs Beat Every Other Source

Analytics tools run JavaScript, so they see human visitors and miss crawlers entirely. Search Console reports aggregated samples. Logs record every request, including:

• Every crawler visit, with timing and frequency.
• Requests to URLs that return errors.
• Requests to URLs nobody links to.
• The exact status code served.
• Bytes transferred, and response time.

Nothing else gives that completeness.

2. The Standard Format

66.249.66.1 - - [18/Aug/2026:14:00:02 +0000] "GET /products/42 HTTP/1.1" 200 18432
  "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Fields: client address, timestamp, request line, status, bytes, referrer, user agent. Some formats add response time, which is worth enabling because it turns the log into a performance record too.

3. Verify Bot Identity — Never Trust the User Agent

The single most important discipline. A user-agent string is trivially forged, so a line claiming to be Googlebot proves nothing.

Verification is forward-confirmed reverse DNS: reverse-resolve the address, check the hostname belongs to the expected domain, then forward-resolve it back to the same address.

import socket

def verified_bot(ip, allowed_suffixes=(".googlebot.com", ".google.com")):
    try:
        host, _, _ = socket.gethostbyaddr(ip)
    except OSError:
        return None
    if not host.endswith(allowed_suffixes):
        return None
    try:
        _, _, addrs = socket.gethostbyname_ex(host)
    except OSError:
        return None
    return host if ip in addrs else None

Both directions must agree, because the reverse record is controlled by the address holder and the forward record by the domain owner — only someone controlling both can produce a consistent pair.

4. What to Measure

The most actionable finding is usually a mismatch: important pages crawled rarely while parameter variants are crawled constantly.

5. Finding Orphan and Phantom URLs

Two comparisons produce immediate value:

Crawled but not in the sitemap — URLs the engine found somewhere you did not declare. Often facets, old URLs, or something accidentally linked.

In the sitemap but never crawled — declared pages the engine is not fetching, which points at discovery or priority problems.

crawled = {l.url for l in log_lines if verified_bot(l.ip)}
declared = set(sitemap_urls)
print("crawled, undeclared:", len(crawled - declared))
print("declared, uncrawled:", len(declared - crawled))

6. Analysing Your Own Crawler's Logs

The technique applies inward. Your own crawler's request log answers questions no application metric does:

• How many requests produced zero extracted records, by domain.
• Which URLs were fetched more than once, indicating a deduplication failure.
• Whether per-domain politeness limits were actually respected.
• How retry attempts are distributed, and whether they succeed.
• Which proxy exits produce which status codes.

That is the structured-logging discipline applied to your own traffic, and it is where crawl inefficiency becomes visible.

7. Practical Notes

1. Enable response time in the log format if it is absent.
2. Verify bots by reverse DNS, and record the verification result.
3. Aggregate, do not eyeball. Logs are too large to read.
4. Watch for your own crawler in someone else's terms — logs are how site operators find you, which is a reason to send an identifying user agent with a contact URL.
5. Treat logs as personal data. IP addresses are personal data in many jurisdictions, so retention limits and access controls apply.

8. Summary

Log file analysis reads server access logs to see exactly which URLs were requested, by whom, and with what result — the only complete record of crawler behaviour, since analytics misses bots entirely.

Verify bot identity with forward-confirmed reverse DNS rather than trusting user agents, look for the mismatch between what you declared and what was crawled, apply the same analysis to your own crawler's logs to find duplicate fetches and zero-record pages, and remember that logs contain IP addresses and are therefore personal data.

Potential Related Glossary

• Web Crawling
• IP Rotation
• Data Wrangling
• Parsing