Scrapeless Wiki

What Is a Web Component? – Definition and How It Works

Informational P2 what is a web component

Learn what custom elements and shadow DOM are, and why web components break ordinary CSS-selector extraction.

A web component is a reusable custom HTML element built from browser-native APIs. For extraction, the important part is that its internals are frequently hidden inside a shadow root that ordinary selectors cannot reach.

1. The Three Pieces

Piece Purpose
Custom Elements Define a new tag with its own behaviour
Shadow DOM An encapsulated subtree, isolated from the page
HTML Templates Inert markup instantiated on demand 
<product-card sku="A1" price="19.99"></product-card>

The tag looks trivial. The content a user sees is generated inside it.

2. Why Extraction Breaks

document.querySelector('product-card .price')     // null — .price is inside the shadow root

querySelector does not cross a shadow boundary. So a page rendered from web components can display prices, titles, and ratings that a conventional CSS-selector extractor cannot see, even though a browser rendered them.

That produces a distinctive symptom: the page looks correct in a screenshot, page.content() shows custom tags with little inside them, and every selector returns nothing.

3. Open Versus Closed Shadow Roots

el.attachShadow({mode: 'open'})     // el.shadowRoot is accessible
el.attachShadow({mode: 'closed'})   // el.shadowRoot is null

Open roots are reachable from script, so extraction is possible by walking into them.

Closed roots are not exposed. The content is rendered but unreachable through the DOM API, which leaves rendered-text extraction or accessibility-tree traversal as the remaining routes.

4. Extracting Through Open Shadow Roots

function deepText(root, out = []) {
  for (const node of root.querySelectorAll('*')) {
    if (node.shadowRoot) deepText(node.shadowRoot, out);
  }
  out.push(root.textContent);
  return out;
}
deepText(document);

A recursive walk that descends into every shadowRoot reaches nested components. Written once, it handles arbitrarily deep composition — which matters because component trees nest.

5. Playwright Pierces Shadow DOM Automatically

A practical convenience worth knowing:

page.locator("product-card >> .price").first.text_content()
page.locator("css=.price").all_text_contents()     # pierces open shadow roots

Playwright's selector engine crosses open shadow boundaries by default, which removes most of this problem. Plain document.querySelector in an evaluated script does not — so the same selector can work through the framework and fail inside page.evaluate.

That inconsistency confuses people regularly. Use the framework's locators rather than hand-rolled DOM queries when components are involved.

6. Slots Complicate the Picture

Content can be passed into a component and rendered through a <slot>:

<product-card><span slot="title">Wireless Keyboard</span></product-card>

Slotted content lives in the light DOM — the ordinary document — so it is reachable by normal selectors, while the component's own chrome is in the shadow root.

So a single component can have some text trivially accessible and other text hidden, which is why partial extraction results are common with components.

7. Practical Guidance

  1. Recognise the symptom: custom tags, empty selectors, correct screenshots.
  2. Use framework locators that pierce shadow DOM rather than raw querySelector.
  3. Walk shadowRoot recursively when scripting the DOM directly.
  4. Check for slotted content in the light DOM before assuming everything is hidden.
  5. Fall back to structured data — a component-heavy page often still carries JSON-LD, which is far easier than DOM archaeology.
  6. Consider the underlying API. A component-rendered page is fetching data from somewhere, and that endpoint is usually a better source than the rendered result.

Point 6 is the strategic answer: heavy component use signals a client-rendered application, and client-rendered applications have APIs.

8. Summary

A web component is a custom element whose internals typically live in a shadow root, which ordinary CSS selectors cannot cross — producing pages that render correctly while every selector returns nothing.

Use locators that pierce open shadow roots, walk shadowRoot recursively when scripting directly, remember slotted content stays in the light DOM, and treat heavy component use as a signal to look for the underlying JSON API instead.