Broken-scraper triage checklist
The problem
A scraper broke and the cause is unknown. The output is empty, partial, duplicated, or stuck, and rerunning it only repeats the failure.
The artefact
A six-section checklist worked top to bottom: selector drift, empty fields, pagination loops, rate limits, login walls, and output checks. Stop at the first section that explains the symptom, fix that one thing, rerun, and check the output again. The checklist and a worked sample report are copied under files on this page.
Input and output sample
Sample report, from the HN defect:
Symptom: 30 rows extracted, submitter column empty on every row, test suite passing. Section 2 (empty fields): one column at 100 percent empty while rows exist means selector problem, not data problem. Section 1 (selector drift): the .hnuser class is gone from the live page, so the selector matched zero elements. Fix: structural selector a[href^="user?id="], plus a test that fails when the selector stops matching.
What can break
- Selector drift: classes change without notice, elements move inside wrappers, iframes, or shadow roots
- Empty fields: one column at 100 percent empty while rows exist, or values that moved between text and attributes
- Pagination loops: the same URL visited twice, next-page controls that never disable
- Rate limits: 429 and 503 responses treated as retry-harder signals instead of slow-down signals
- Login walls: a redirect to a login form returning HTTP 200 with no data
- Output checks skipped: row counts, types, and required columns never compared against the last good run
The fix or design choice
The checklist orders the checks so the cheapest explanation wins first, and the sample report shows the path taken on a real defect: section 2 caught the symptom, section 1 named the cause. Green tests against a fixture prove the code matches the fixture. Only a live-page check proves it matches the site.
Verification log
- 2026-11-19: checklist and sample report reviewed against the verified HN defect facts (2026-11-12, 30 rows, 6 of 6 tests)
- No code in this artefact. Read CHECKLIST.md and sample-triage-report.md under files
Verified 2026-11-19
Files
Price shape
Paid scraper diagnostic $150 to $300, triage only, with a written acceptance check: selectors that still match, a sample CSV, and a short failure log. If the sample matches your columns, the full scraper quote follows in writing.
Start here
Email the target URL and describe the symptom: empty fields, partial rows, duplicates, or a stuck run. I reply with the diagnostic scope and price.