Why Does My Selector Keep Breaking? – Causes and Durable Fixes
Learn why extraction selectors break: generated class names, layout drift, conditional markup, rendering timing — and how to write ones that last.
Selectors break because they encode assumptions about markup that the site never promised to keep. The fix is choosing anchors the site has a reason to keep stable.
1. The Five Causes
| Cause | Symptom |
|---|---|
| Generated class names | Worked yesterday, matches nothing today, class looks like random characters |
| Structural drift | A wrapper element added, breaking a long child chain |
| Conditional markup | Works on most pages, fails on some — the field is optional or rendered differently |
| Timing | Selector runs before the content arrives |
| A/B variants | Two markups exist; you only ever inspected one |
2. Generated Class Names
Build tools produce hashed class names that change whenever styles change. A selector anchored on one is guaranteed to break, usually on the next deploy.
Anchor instead on attributes with semantic meaning — id, data-*, name, role, aria-label — because those exist for the site's own code and tests, so it has a reason to keep them.
3. Structural Drift
A chain like div > div > div:nth-child(2) > span encodes the entire layout. One inserted wrapper invalidates it.
Prefer a short relative selector from a stable container: find the element whose identity is meaningful, then search within it. If a component moves, only the container reference needs updating.
4. Conditional Markup
A field that is absent, or rendered differently, on some pages produces intermittent failures that look random.
Write extraction so a missing element yields an explicit None rather than an exception, and test against a sample of pages that includes edge cases — sold-out items, listings without ratings, records in another language.
5. Timing
On client-rendered pages a selector can run before the content exists. This fails intermittently and appears to depend on network speed, because it does.
Wait on a condition that proves the content arrived — the element itself, a network response, or a minimum count — rather than on a fixed delay.
6. Writing Durable Selectors
- Prefer
data-*attributes and ids. - Anchor on text where the label is stable — this is where XPath earns its place.
- Keep chains short and relative to a meaningful container.
- Avoid positional indexes unless order is genuinely guaranteed.
- Match on attribute prefixes where identifiers are partly generated.
- Extract from embedded JSON or a JSON endpoint instead, where one exists — data contracts change far less than layouts.
7. Detecting Breakage Early
Silent breakage is worse than loud breakage. Assert on every run:
- An expected minimum record count.
- Required fields non-empty on a defined share of records.
- Value types and plausible ranges.
A run that returns zero records should fail loudly rather than write an empty file that looks like a quiet day.
8. Summary
Selectors break because they depend on incidental markup rather than on anchors the site maintains deliberately. Generated classes, long structural chains, and positional indexes are the three most fragile choices.
Anchor on semantic attributes, keep selectors short and relative, prefer structured sources where they exist, and assert on output so a break is announced rather than silently absorbed.