Chris Butcher, Python automation

URL and column-spec scraper

The problem

You need one site scraped into a spreadsheet, with the columns named up front and proof that no column came back silently empty.

The artefact

A small CLI that takes a URL and a JSON column spec, fetches the page with Playwright, extracts one row per element matched by the spec row selector, and writes CSV plus a short run log. The log records rows found, empty-field counts per column, and failures. The artefact README, spec example, and source are copied under files on this page.

Input and output sample

Input spec:

{
  "row_selector": ".book",
  "columns": {
    "title": {"selector": ".title", "required": true},
    "price": {"selector": ".price"},
    "link": {"selector": "a", "attribute": "href"}
  }
}

Output CSV, from the bundled fixture with three rows:

title,price,link
Alpha Title,10.50,/books/alpha
Beta Title,12.00,/books/beta
Gamma Title,,/books/gamma

Output run log:

rows found: 3
empty fields per column:
  title: 0
  price: 1
  link: 0
failures:
  none

What can break

The fix or design choice

A column marked required fails the run when any row leaves it empty, so a silent selector break shows up as a failed run with exit code 2 instead of a blank column shipped downstream. Exit codes: 0 success, 2 required column empty, 1 other failure. Before pointing the CLI at a real site, each selector gets verified against the live DOM.

Verification log

Verified 2026-11-19

Files

Price shape

Scrapers from $300 fixed. A paid diagnostic at $150 to $300 covers triage first: selectors that still match, a sample CSV, and a failure log, with the acceptance check in writing before work starts.

Start here

Email the target URL and the exact columns you need. I reply with the fixed quote and the first sample plan.