# Broken-scraper triage checklist

Work top to bottom. Stop at the first section that explains the
symptom, fix that one thing, rerun, and check the output again.
Most broken scrapers fail in one of these six places.

## 1. Selector drift

- [ ] Open the live page and inspect the element in the DOM today,
      not from memory and not from an old fixture.
- [ ] Confirm each selector matches the count you expect. A
      selector that returns zero elements is drift until proven
      otherwise.
- [ ] Prefer stable hooks in this order: data attributes, ids,
      structural attribute selectors, then classes. Classes are
      presentation and change without notice.
- [ ] Check whether the element moved inside a new wrapper, an
      iframe, or a shadow root.
- [ ] After fixing, add a test that fails when the selector stops
      matching, so the next drift is loud.

## 2. Empty fields

- [ ] Count empty values per column in the output. One column at
      100 percent empty while rows exist is a selector problem,
      not a data problem.
- [ ] Distinguish "field absent on this row" from "field absent
      on every row". The first can be real data. The second is a
      break.
- [ ] Check whether the value moved from text content to an
      attribute, or the other way.
- [ ] Check timing: the field may render after the row does. Wait
      for the field selector, not a fixed sleep.
- [ ] Mark columns that must never be empty as required and fail
      the run when they are. A failed run beats a blank column
      shipped downstream.

## 3. Pagination loops

- [ ] Log the URL of every page visited. A loop shows the same
      URL twice.
- [ ] Confirm the next-page control disappears or disables on the
      last page. If it stays enabled, stop on an empty page or a
      repeated first row instead.
- [ ] Cap total pages in code. A cap turns an infinite loop into
      a visible failure.
- [ ] Watch for query parameters that reset: page 2 linking back
      to page 1 with a different sort or filter parameter.
- [ ] Dedupe rows on a stable key before writing output, and
      report how many duplicates were dropped.

## 4. Rate limits

- [ ] Read the status codes. 429 and 503 with a Retry-After
      header mean slow down, not retry harder.
- [ ] Add delay between requests and honor Retry-After. Back off
      exponentially on repeated limits.
- [ ] Reduce concurrency before adding proxies. One polite
      session usually beats five fast ones.
- [ ] Check whether the block is IP based, session based, or
      fingerprint based by testing one request from a clean
      session.
- [ ] Record the limit you found in the run log so the next run
      starts inside it.

## 5. Login walls

- [ ] Confirm the page you scraped is the page you think it is.
      A redirect to a login form returns HTTP 200 with no data.
- [ ] Check for consent banners, region gates, and bot checks
      that render instead of content.
- [ ] If the data needs an account, use a real session and store
      the session state outside the repo. Never commit
      credentials.
- [ ] Assert on a marker that only exists on the real page, such
      as a logout link or the target heading, and fail the run
      when it is missing.
- [ ] Screenshot the page on failure. One screenshot settles
      "login wall or empty result" faster than any log line.

## 6. Output checks

- [ ] Row count in range: compare against the last good run and
      flag swings over an agreed threshold.
- [ ] Types enforced: numbers parse as numbers, dates parse as
      dates, ids are unique.
- [ ] Spot-check first, middle, and last rows against the live
      page by eye.
- [ ] Required columns non-empty on every row.
- [ ] The run log states rows found, empty counts, pages visited,
      and failures, so the next person can see what happened
      without rerunning anything.
