How to Build a RAG Pipeline From Web Data – A Practical Guide
Build a RAG pipeline whose source is the open web: collection, extraction, chunking, embedding, refresh, and the failure modes unique to web data.
A RAG pipeline over your own documents is mostly a formatting exercise. A pipeline over the open web adds three problems the internal case never has: pages change without telling you, most of a page is not content, and collection can fail intermittently at scale.
1. Decide the Corpus Before the Code
Start from the questions the system must answer, then work back to the smallest set of pages that contains those answers.
- Named sources — a fixed list of sites, crawled on a schedule. Predictable and easy to refresh.
- Query-driven — search at question time, fetch the top results, use them once. Always current, slower per question.
- Hybrid — a standing index for stable material, live search for anything time-sensitive.
Most production systems end up hybrid, because a nightly index cannot answer about this morning and live fetching every question is slow.
2. Collect the Pages
Collection is a pipeline stage with its own reliability characteristics, not a one-line fetch.
| Requirement | Why |
|---|---|
| JavaScript execution where needed | Much of the web assembles content client-side; a plain fetch returns an empty shell |
| Bounded concurrency | Politeness and stability both argue against unbounded parallel requests |
| Stored raw response | So extraction can be re-run without re-fetching |
| Recorded fetch time and status | Freshness and refresh decisions depend on it |
Store the raw HTML and the extracted text separately. When extraction improves, the corpus can be rebuilt without touching the network.
3. Extract Content, Not Pages
This is the step that most affects answer quality. A saved page is mostly navigation, cookie notices, related-article rails, and footers.
- Convert to clean text or Markdown, preserving headings, lists, and tables.
- Drop boilerplate that repeats across pages of the same site — it retrieves as noise on every query.
- Keep the title and canonical URL with the text, because citation depends on them.
- Reject pages whose extracted text is implausibly short; that usually signals a challenge page or an empty shell rather than a real document.
4. Chunk on Structure
Split on headings and section boundaries rather than a fixed character count, so a definition stays with its example and a table stays with its header. Carry the page title and section heading into each chunk's stored text — retrieval often surfaces a passage whose subject is only stated in the heading above it.
5. Embed, Store, and Filter
Embed each chunk and store it with its vector, text, and metadata. For web corpora the metadata that earns its keep is source_domain, fetched_at, language, and canonical_url.
Those fields power the filters that make answers trustworthy: restrict to the domains you trust, restrict to documents fetched within a window, and exclude languages the model should not mix in.
6. Plan the Refresh
Web corpora rot silently. A page that changed last week still retrieves, still looks authoritative, and is now wrong.
- Re-fetch on a schedule proportional to how fast the source changes.
- Compare a content hash before re-embedding; unchanged pages cost nothing.
- Delete chunks whose source page is gone, rather than leaving them retrievable.
- Surface
fetched_atin the answer so a reader can judge freshness.
7. Failure Modes Specific to Web Data
- Challenge and consent pages indexed as content. They extract cleanly and say nothing. Filter on length and on telltale phrasing.
- Near-duplicate pages. Syndicated articles crowd out diverse sources; deduplicate on content hash or embedding proximity before storage.
- Mixed languages. Without a language filter, retrieval returns a correct answer in the wrong language.
- Boilerplate drift. A site redesign changes what the extractor keeps; re-check extraction quality after any large re-crawl.
8. Summary
A web-sourced RAG pipeline is a collection problem first and a retrieval problem second. Decide the corpus, collect reliably with JavaScript where required, extract content rather than pages, chunk on structure, and keep freshness metadata you can filter and display.
Do that and the retrieval half becomes ordinary. Skip it and no amount of prompt engineering will make the answers trustworthy, because the index will be full of navigation menus.