Web Scraping With Go – Tools and Patterns
Learn how to scrape in Go: the standard HTTP client, goquery for parsing, bounded concurrency with goroutines, and when a browser is needed.
Go suits collection work well: the standard library covers HTTP thoroughly, concurrency is built into the language, and a compiled binary deploys without a runtime.
1. Fetching
client := &http.Client{Timeout: 30 * time.Second}
req, _ := http.NewRequest("GET", url, nil)
req.Header.Set("User-Agent", "example-collector/1.0")
resp, err := client.Do(req)
if err != nil { return err }
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("unexpected status %d", resp.StatusCode)
}
Two habits matter: always set a timeout on the client, and always close the body — a leaked body exhausts connections in a long-running collector.
2. Parsing
goquery provides a jQuery-style API over the standard HTML parser:
doc, err := goquery.NewDocumentFromReader(resp.Body)
doc.Find("article.product").Each(func(i int, s *goquery.Selection) {
title := strings.TrimSpace(s.Find("h3").Text())
price := strings.TrimSpace(s.Find(".price").Text())
})
For XPath, htmlquery covers cases where text anchors or upward traversal are needed.
3. Bounded Concurrency
Goroutines make unbounded concurrency dangerously easy. Bound it explicitly:
sem := make(chan struct{}, 8)
var wg sync.WaitGroup
for _, u := range urls {
wg.Add(1)
go func(u string) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
collect(u)
}(u)
}
wg.Wait()
A buffered channel as a semaphore is the idiomatic limiter, and per-host limits matter more than the global one.
4. Reusing Connections
The default transport pools connections, but only if response bodies are fully read and closed. A body that is closed without being drained forfeits reuse.
For many requests to one host, raising MaxIdleConnsPerHost on a custom transport makes a measurable difference.
5. Context for Cancellation
context propagates deadlines and cancellation through a call tree. Attaching one to each request means a whole collection can be stopped cleanly rather than waiting for every in-flight call to time out individually.
6. When a Browser Is Needed
Go does not execute JavaScript. For client-rendered pages the options are calling the endpoint the page uses, or driving a browser through a CDP library such as chromedp.
The order matters: look for the endpoint first, since a browser is far more expensive per page.
7. Practical Notes
- Set timeouts on the client and on contexts.
- Read and close bodies without exception.
- Limit concurrency per host, not only globally.
- Handle encodings explicitly where pages are not UTF-8.
- Write JSON Lines for streaming output.
8. Summary
Go handles collection with the standard HTTP client, goquery for parsing, and goroutines with an explicit semaphore for concurrency, deployed as a single binary.
Its strengths are throughput and operational simplicity; its constraint is the absence of JavaScript execution, which means finding the underlying endpoint or driving a browser for client-rendered pages.