Web Scraping With Rust – Tools and Patterns

Informational P2 web scraping with rust

Learn how to scrape in Rust: reqwest for fetching, scraper for parsing, bounded async concurrency, and where the strictness pays off.

Rust suits collectors that must run continuously without supervision: memory safety without a garbage collector, explicit error handling, and a single deployable binary.

1. Fetching With reqwest

let client = reqwest::Client::builder()
    .timeout(std::time::Duration::from_secs(30))
    .user_agent("example-collector/1.0")
    .build()?;

let response = client.get(url).send().await?;
if !response.status().is_success() {
    anyhow::bail!("unexpected status {}", response.status());
}
let body = response.text().await?;

The client holds the connection pool and should be built once and shared, not created per request.

2. Parsing With scraper

use scraper::{Html, Selector};

let document = Html::parse_document(&body);
let card = Selector::parse("article.product").unwrap();
let title = Selector::parse("h3").unwrap();

for element in document.select(&card) {
    let name = element.select(&title).next()
        .map(|e| e.text().collect::<String>().trim().to_string());
}

Selectors are compiled once and reused; parsing them inside a loop is a common and needless cost.

3. Errors Are Explicit

Rust has no exceptions. Every fallible operation returns a Result that must be handled, so the "element might be missing" case cannot be ignored by accident.

That is the language's main contribution to collection code: the failure modes that silently produce empty records elsewhere are compile-time obligations here.

4. Bounded Async Concurrency

use futures::stream::{self, StreamExt};

let results: Vec<_> = stream::iter(urls)
    .map(|url| { let c = client.clone(); async move { collect(&c, url).await } })
    .buffer_unordered(8)
    .collect()
    .await;

buffer_unordered is the idiomatic limiter. As always the meaningful limit is per host rather than global.

5. Where the Strictness Pays Off

Long-running collectors, where leaks and unhandled cases accumulate.
High throughput, where per-request overhead matters.
Constrained environments, where a small static binary is an advantage.
Correctness-sensitive extraction, where a missing field must be handled rather than defaulted.

6. Where It Costs

Development is slower than in a dynamic language, particularly for exploratory work where the shape of a page is still being learned.

A common division is prototyping extraction in Python and porting the settled logic to Rust once it is stable and the volume justifies it.

7. Rendered Pages

Rust does not execute JavaScript. Options are calling the page's underlying endpoint, or driving a browser over CDP through a crate such as chromiumoxide.

The endpoint should always be checked first; browser control from Rust works and is more effort than in ecosystems built around it.

8. Summary

Rust collects with reqwest and scraper, bounded async concurrency, and compile-time handling of every fallible step, deployed as one binary.

It suits long-running, high-throughput collectors where correctness and resource use matter, and it is heavier than necessary for exploratory work — which is why prototype-then-port is the common pattern.