Chris Butcher, Python automation

HN scraper defect: passing tests, empty column

The problem

A scraper can pass every test and still ship broken data. The Hacker News front-page scraper extracted the top 30 stories with rank, title, domain, points, submitter, and comment count. The test suite passed. The submitter column in the output CSV was empty on every row.

The artefact

The full pipeline, its CSV output, and the verification log live on the artefact store. The triage write-up is copied here as sample-triage-report.md.

Input and output sample

Input: the live Hacker News front page, top 30 stories.

Output before the fix: 30 rows, submitter empty on all 30. Output after the fix: 30 rows, submitter populated on every row.

What can break

The fix

Replaced the class selector with a structural attribute selector, a[href^="user?id="], which matches the submitter link by its URL pattern instead of a presentation class. Added tests that fail when the submitter selector stops matching, so the next drift breaks the suite instead of the output.

Verification log

Verified 2026-11-19

Files

Price shape

This is the failure a paid diagnostic is built to find. Triage runs $150 to $300 with a written acceptance check: selectors that still match, a sample CSV, and a short failure log. Full scraper builds start at $300.

Start here

Send the target URL and the exact columns you need. I reply with the fixed quote and the first sample plan.