Web Scraping With JavaScript – Tools and Patterns

Learn how to scrape with Node.js: fetching, parsing with Cheerio, driving a browser when needed, concurrency control, and writing clean output.

JavaScript is a natural fit for scraping because the ecosystem's browser tooling is first-class and the language already matches the environment pages run in. Most work divides into two paths: fetch and parse, or drive a browser.

1. Fetch and Parse

For server-rendered pages, a plain request plus a parser is fastest.

const res = await fetch(url, { headers: { 'accept-language': 'en-US,en;q=0.9' } });
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const html = await res.text();

fetch is built into modern Node. Parsing is usually Cheerio, which offers a jQuery-style API over a parsed document without running scripts or rendering.

import * as cheerio from 'cheerio';

const $ = cheerio.load(html);
$('article.product').each((_, el) => {
  const title = $(el).find('h3').text().trim();
  const price = $(el).find('.price').text().trim();
});

2. When a Browser Is Required

If the served HTML lacks the content, scripts are building it. Then a browser library — Playwright or Puppeteer — is the tool.

The third row is the one people skip. Watching the network panel often reveals a clean JSON API, which is faster and more stable than parsing rendered markup.

3. Concurrency Without Overload

Node's asynchrony makes it trivially easy to launch hundreds of simultaneous requests, and equally easy to be rate limited or blocked. Bound it explicitly with a small pool:

async function pool(items, limit, worker) {
  const results = [];
  const active = new Set();
  for (const item of items) {
    const p = worker(item).then(r => { active.delete(p); return r; });
    active.add(p);
    results.push(p);
    if (active.size >= limit) await Promise.race(active);
  }
  return Promise.all(results);
}

A small limit with steady pacing outperforms a large one that triggers defensive responses.

4. Cleaning Output

Text from markup carries whitespace, non-breaking spaces, and stray newlines. Normalise at extraction:

const clean = s => (typeof s === 'string' ? s.replace(/\s+/g, ' ').trim() : s);

Emit one object per record with explicit null for missing fields, and write JSON Lines rather than one large array so the file can be appended and streamed.

5. Practical Requirements

• Timeouts on every request; an AbortController bounds fetch.
• Status checks — fetch does not throw on 4xx or 5xx, so check res.ok explicitly.
• Sessions — preserve cookies when a flow spans requests.
• Politeness — honour robots.txt and keep the request rate reasonable.

6. Common Mistakes

• Assuming fetch throws on errors. It only rejects on network failure.
• Reaching for a browser first. It is orders of magnitude more expensive per page than a fetch.
• Regex over HTML. Fine for one attribute, fragile for anything nested.
• Unbounded concurrency. The fastest way to be blocked.

7. Real-World Examples

• A catalogue crawl over paginated server-rendered listings using fetch and Cheerio.
• A price check on a client-rendered product page using a browser session.
• A search collector that calls the page's own JSON endpoint discovered in the network panel.
• A monitoring job that fetches on a schedule and writes JSON Lines for downstream comparison.

8. Summary

Scraping with JavaScript means fetching and parsing where possible, and driving a browser only where scripts assemble the content. Cheerio covers parsing; Playwright or Puppeteer covers rendering.

The habits that matter are unglamorous: check status explicitly, set timeouts, bound concurrency, normalise text at extraction, and look for an underlying JSON endpoint before writing a single selector.

Potential Related Glossary

• Parsing
• Headless Browser
• Web Crawling
• Axios