# URL plus column-spec scraper

A small CLI that takes a URL and a JSON column spec, fetches the page
with Playwright, extracts one row per element matched by the spec row
selector, and writes CSV plus a short run log. The log records rows
found, empty-field counts per column, and failures. A column marked
required fails the run when any row leaves it empty, so a silent
selector break shows up as a failed run instead of a blank column.

## Run

From this directory, using the shared venv:

```bash
cd ~/freelance-engine/projects/dev-artefacts/url-column-scraper
~/freelance-engine/.venv/bin/python scraper.py \
  --url "file://$PWD/fixture.html" \
  --spec spec.example.json \
  --out output.csv \
  --log run-log.txt \
  --no-proxy
```

For a live http(s) page, drop --no-proxy. The CLI routes http(s)
requests through the local relay proxy at http://127.0.0.1:18082 and
ignores certificate errors, matching the VM egress setup.

## Spec format

```json
{
  "row_selector": ".book",
  "columns": {
    "title": {"selector": ".title", "required": true},
    "price": {"selector": ".price"},
    "link": {"selector": "a", "attribute": "href"}
  }
}
```

A flat mapping of column name to selector string or object also
works when the row selector is passed with --row-selector.

## Sample input and output

Input: fixture.html (three .book rows, the third has no price) with
spec.example.json.

Output output.csv:

```csv
title,price,link
Alpha Title,10.50,/books/alpha
Beta Title,12.00,/books/beta
Gamma Title,,/books/gamma
```

Output run-log.txt:

```text
url: file:///home/hatch/freelance-engine/projects/dev-artefacts/url-column-scraper/fixture.html
rows found: 3
empty fields per column:
  title: 0
  price: 1
  link: 0
failures:
  none
```

Exit codes: 0 success, 2 required column empty, 1 other failure.

## Tests

```bash
~/freelance-engine/.venv/bin/python -m pytest tests/ -q
```

Tests run offline against the bundled fixture: parsing, empty-field
counts, the required-field guard both ways, CSV output, and a full
CLI run.

## Verification log

- 2026-11-19: test suite run from a clean shell, 6 passed.
- Fixture and spec are bundled, so the suite needs no network.
- Live-page selector checks are out of scope for the bundled tests.
  Before pointing the CLI at a real site, verify each selector
  against the live DOM and keep the required flag on columns that
  must never come back empty.
