What Is Colly? – Definition and How It Works

Learn what the Colly scraping framework for Go provides, how its callback model works, and where it fits.

Colly is a scraping framework for Go built around a Collector and a set of callbacks. It handles the crawl loop — requests, HTML dispatch, link following, rate limiting — while you write the extraction.

1. The Shape of a Colly Program

c := colly.NewCollector(
    colly.AllowedDomains("example.com"),
    colly.MaxDepth(2),
)

c.Limit(&colly.LimitRule{
    DomainGlob:  "*",
    Parallelism: 4,
    Delay:       200 * time.Millisecond,
    RandomDelay: 200 * time.Millisecond,
})

c.OnHTML("h2.title a[href]", func(e *colly.HTMLElement) {
    fmt.Println(e.Text, e.Request.AbsoluteURL(e.Attr("href")))
    e.Request.Visit(e.Attr("href"))
})

c.OnError(func(r *colly.Response, err error) {
    log.Printf("error %s: %v", r.Request.URL, err)
})

c.Visit("https://example.com/")
c.Wait()

Three things carry the design: CSS selectors dispatch to callbacks, AbsoluteURL resolves relative links correctly, and Limit expresses politeness declaratively rather than as sleeps.

2. The Callback Set

OnRequest is where headers are set per request; OnScraped is where a per-page record is finalised, which is the natural place to write output exactly once.

3. Why Go Suits Crawling

Crawling is I/O-bound concurrency, which is what goroutines are for. A Colly collector running with parallelism configured uses very little memory per in-flight request compared with a thread-per-request model, and Go's single static binary makes deployment trivial.

The practical consequence: high-concurrency crawling of static HTML on modest hardware, with no runtime to install.

4. What It Does Not Do

JavaScript. Colly is an HTTP client with an HTML parser. Pages that render client-side return the shell. That is the main limitation, and it is architectural — the fix is a separate rendering step or a different tool.

Automatic anti-bot handling. It sets a user agent and honours limits; it does not present a browser fingerprint.

Built-in storage. Persistence is yours to write, though a pluggable storage interface exists for the visited-URL set and cookies.

5. Persistent Queues and Storage

By default the visited set is in memory, so a restart re-crawls everything. The storage interface allows a backend — Redis or SQLite, for instance — which makes a crawl resumable and lets multiple processes share one frontier.

For any crawl large enough to be interrupted, this is not optional. It is also the single change that turns a Colly script into an operable crawler.

6. Colly Compared

Colly and Scrapy occupy the same niche; the choice is usually the surrounding language and whether Scrapy's middleware ecosystem is worth its overhead.

7. Practical Notes

• Always set Limit. The default is unthrottled, which is impolite and gets you blocked.
• Use RandomDelay so the request cadence is not perfectly periodic.
• Set AllowedDomains, or one bad link takes the crawl off-site.
• Clone collectors (c.Clone()) for a differently configured phase, such as detail pages with other limits.
• Handle OnError explicitly. A silent error handler produces a crawl that reports success while collecting nothing.
• Watch callback concurrency. Callbacks run on multiple goroutines, so shared state needs synchronisation — a mutex or a channel to a single writer.

That last point is the most common Colly bug: appending to a shared slice from OnHTML without a lock is a data race that usually manifests as lost records rather than a crash.

8. Summary

Colly is a Go scraping framework with a Collector and selector-driven callbacks, handling the crawl loop, link resolution, and declarative rate limiting while you write extraction.

It is fast, concurrent, and deploys as a single binary, but does not execute JavaScript and provides no storage by default — so add a persistent backend for resumability, always configure limits, and synchronise any state shared between callbacks.

Potential Related Glossary

• Web Crawling
• Concurrency vs Parallelism
• Parsing
• Headless Browser