Crawl

Collect the URLs behind one start page from the terminal, and each page as HTML or Markdown when you ask for it.

Crawl is New, and zenrows crawl ships in the Zenrows CLI from version 1.5.0. Any active Zenrows API key can use it. Install or update with npx -y @zenrows/cli.

zenrows crawl starts from one URL and returns the URLs it finds from there. A crawl runs in the background: create returns at once, then wait, results, content, and download read it.

zenrows crawl <create|get|list|results|content|download|stop|wait> [flags]

depth is optional. Leave it out and the crawl follows links until max_items or max_pages stops it, or nothing is left to open. Both limits default to 100 and cap at 100,000. The crawl stays on the start URL’s domain, subdomains included.

Examples

# Start a crawl and wait for it to finish
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --depth 1 --include-patterns /product/ --follow

# No depth limit. Stop after 20 fetched pages
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --max-pages 20

# Collect the URLs
zenrows crawl results <crawl-id> --out urls.jsonl

# Also fetch each kept page's HTML
zenrows crawl create https://www.scrapingcourse.com/ecommerce/ --output-format html --max-pages 20 --follow
zenrows crawl content <crawl-id> <content-id>
zenrows crawl download <crawl-id>

# Stop an in-flight crawl
zenrows crawl stop <crawl-id>

--include-patterns chooses which links Crawl opens, not only which URLs it returns. A homepage crawl with only /product/ skips listing paths that lack that text, such as /page/. See Crawl setup.

--follow and wait stop waiting after --timeout seconds (default 600). The crawl keeps running. Ctrl-C stops the wait, not the crawl.

Flags

--json prints structured output on every subcommand. The rest are subcommand-specific:

Subcommand	Flags
create <url>	--depth <n> (1 to 100,000; omit for no depth limit), --max-items <n>, --max-pages <n> (API default 100 each, cap 100,000; --max-pages is also capped by the local policy max_pages_per_run, default 1,000), --include-patterns <p> (repeatable), --exclude-patterns <p> (repeatable), --output-format <html|markdown>, --discovery links, --follow, --timeout <s>, --idempotency-key <key>, --no-signup
get <id>	--cursor <c>, --limit <n> (up to 10,000)
list	--cursor <c>, --limit <n> (up to 100)
results <id>	--limit <n> (page size), --out <file>
content <id> <content_id>	--out <file>. content_id is a content id or a result’s content_url
download <id>	--out <file> (default <id>.jsonl)
wait <id>	--timeout <s>
stop <id>

results prints the URLs kept so far. They are partial while the crawl runs. After a stop, the pages already in flight can still add URLs for up to 10 minutes. results then says to read them again later. download is partial at the same times, and says so. Call wait first to read every result of a crawl that runs to its end.

Each page a crawl fetches is one Fetch (formerly, Universal Scraper API) request. See Crawl pricing.