Web Scraping With Ruby

Learn how to scrape with Ruby: HTTP clients, Nokogiri parsing, browser control with Ferrum, and how Ruby compares to Python for this work.

Ruby has a compact, high-quality scraping toolchain. It is smaller than Python's, but the pieces are mature — and if your application is already Rails, extracting in Ruby avoids a second language in the pipeline entirely.

1. The Stack

Nokogiri is the centre of gravity. It is fast, complete, and essentially universal in Ruby scraping.

2. Fetching

require 'faraday'

conn = Faraday.new(headers: { 'User-Agent' => UA }) do |f|
  f.request :retry, max: 3, interval: 0.5, backoff_factor: 2
  f.options.timeout = 10
end

response = conn.get('https://example.com/products')

Faraday's middleware model is its strength — retries, logging, instrumentation and error handling compose as layers rather than accumulating at call sites. HTTParty is simpler for quick work; Net::HTTP needs no gem at all.

3. Parsing With Nokogiri

require 'nokogiri'

doc = Nokogiri::HTML(response.body)

title = doc.at_css('h1')&.text&.strip

products = doc.css('.product').map do |node|
  {
    name:  node.at_css('.name')&.text&.strip,
    price: node.at_css('.price')&.text&.strip,
    url:   node.at_css('a')&.[]('href')
  }
end

Nokogiri supports both CSS and XPath, which is a real advantage over parsers offering only one:

doc.xpath('//div[@class="product"][position() <= 10]')
doc.css('.product:first-child')

XPath handles things CSS cannot express — selecting by text content, walking to ancestors, applying positional predicates.

Note &. throughout. at_css returns nil when nothing matches, and Ruby's safe navigation operator is what keeps that from becoming a NoMethodError three frames away.

4. Browsers

Ferrum is the idiomatic choice — a Ruby CDP client, no WebDriver binary:

require 'ferrum'

browser = Ferrum::Browser.new(headless: true)
browser.goto('https://example.com')
browser.at_css('#load-more')&.click
browser.network.wait_for_idle
doc = Nokogiri::HTML(browser.body)
browser.quit

Fetching the body and handing it to Nokogiri is the common pattern: use the browser to produce the HTML, use Nokogiri to parse it.

Selenium and the Playwright Ruby client are also available. As always, check first whether the page's own JSON endpoint makes the browser unnecessary.

5. Mechanize for Sessions

For multi-step flows — logging in, submitting forms, following links — Mechanize maintains cookies and form state:

agent = Mechanize.new
page  = agent.get('https://example.com/login')
form  = page.forms.first
form.username = user
form.password = pass
dashboard = agent.submit(form)

It handles cookies, redirects and form serialisation, which is a lot of tedious code avoided for server-rendered sites. It does not run JavaScript.

6. Ruby vs. Python for Scraping

Two gaps matter in practice.

Concurrency. Ruby's async ecosystem is less developed, and scraping is I/O-bound work where that matters. Threads help, and the tooling is thinner than asyncio plus httpx.

TLS impersonation. Ruby has no widely used equivalent to curl_cffi. Against targets that fingerprint clients at the TLS layer, this is a hard limitation rather than an inconvenience — the usual answer is a real browser or an external scraping API.

Ruby's clearest case is an existing Rails application. Keeping extraction in the same codebase, with the same models, jobs and deployment, usually beats introducing Python for one component.

7. Practices

• Use &. consistently. Missing nodes return nil, and the error otherwise surfaces far from its cause.
• Assert on counts. raise "no products at #{url}" if products.empty? is the difference between noticing a redesign and storing empty rows for a month.
• Anchor on stable selectors. [data-testid] over generated class names.
• Save fixtures and parse them in tests, so you can tell a site change from a code change.
• Use ActiveJob or Sidekiq for scheduling if you are already in Rails, rather than building a runner.
• Respect the target. Delays, concurrency caps, and backoff on 429.

8. Summary

Ruby scraping centres on Nokogiri for parsing, with Faraday or HTTParty for fetching, Ferrum for browser control, and Mechanize for session-based flows. Nokogiri supporting both CSS and XPath in one library is a genuine convenience.

The ecosystem is smaller than Python's, and two gaps are real: weaker async concurrency for I/O-bound work, and no mature TLS impersonation library, which limits what is reachable against fingerprinting targets.

Where Ruby wins is context. In an existing Rails codebase, scraping in Ruby keeps one language, one deployment and one set of models — and that usually outweighs a larger ecosystem in another language.

Potential Related Glossary

• Parsing
• Web Crawling
• Unstructured Data
• Data Wrangling