Evals

A reproducible benchmark bundled with the Zenrows CLI, with explicit pass or fail criteria.

An eval is a scripted benchmark against a known target. Run it to check that a command still behaves as expected, and get a written report back.

zenrows eval list
zenrows eval run <name>

Command	Does
zenrows eval list	List all evals
zenrows eval run <name>	Run an eval and write a report
zenrows eval report <name>	Print a previous eval’s report
zenrows eval explain <name>	Show an eval’s full docs
zenrows eval install <name>	Copy an eval into .zenrows/
zenrows eval update [name]	Refresh installed eval(s)
zenrows eval remove <name>	Remove an installed eval

Catalog

Eval	Description
protected-fetch-smoke	fetch against a known-good URL, expects HTTP 2xx and content
extract-smoke	extract --autoparse against a known-good ecommerce demo page
batch-spec-validation	A JSONL spec parses cleanly and estimates cost, no cloud call

Each run writes input.json, results.json, report.md, failures.jsonl, cost.json, and traces/ under .zenrows/evals/<run-id>/.