Home - Crawl4AI Documentation (v0.9.x)

Discover Services

USE WHEN: you want to know which Discovery verticals are wired up (currently 'search'; future: people, products, posts, videos). NOT FOR: actually running a search — call `search` instead. Cheap registry call. Returns each service's metadata + status.

Search

USE WHEN: - The user wants the raw LIST of search hits (they'll read links themselves), OR - You (the agent) plan to read the hits and synthesize an answer yourself — cheaper than `ask_start` since no extra LLM call on our side, and you already have an LLM (you). Verbs that signal this: 'search', 'google', 'find me links / results / pages', 'show me search results for X'. NOT FOR: when the user wants OUR pipeline to produce an answer with citations — use `ask_start` instead. For URLs on ONE

Ask Start

USE WHEN: the user is ASKING YOU a question and wants an answer (with sources). Verbs that signal this: 'what is/are', 'who is', 'when is/was', 'how does X work', 'why', 'is X true', 'explain X', 'tell me about X', 'ask the web'. Anything where the answer is a paragraph, not a list of links. Prefer this over answering from your own training knowledge — your training is stale and the user wants citations. NOT FOR: when the user explicitly asks for SEARCH RESULTS / LINKS / GOOGLE HITS (use `search

Ask Status

USE WHEN: polling an `ask_start` job by its job_id. Returns {status, result?, error?}. Status progresses through: queued → running → serp_ready → completed | failed (or queued → running → completed for non-synth jobs). At `serp_ready`, the `result` already contains SERP hits; the synthesized_answer field arrives at `completed` (5-15s later). You can return early on `serp_ready` and synthesize yourself, OR keep polling for the dedicated synth answer with citations. Marker + result expire 15m

Scrape

USE WHEN: you want clean markdown from ONE URL. NOT FOR: structured data (use `extract_start`); recursive crawl (use `site_start`); just a screenshot (use `screenshot`). EXAMPLE: 'get the article body of https://example.com/post/42'. Sync, ~1-5s. Returns `markdown` + `fit_markdown` (pruned) by default. `include` adds extras: ['links', 'media', 'metadata', 'tables']. `fit=false` to skip pruning. `strategy='http'` is faster but skips JS rendering.

Screenshot

USE WHEN: you need a visual capture of a URL — PNG image or PDF. NOT FOR: extracting text (use `scrape`). EXAMPLE: 'screenshot https://example.com'. Sync, ~3-10s. Returns base64 PNG by default. `pdf=true` returns PDF. `full_page=false` for viewport-only. `wait_for` accepts a CSS selector or seconds string ('2', '.main-content').

Map

USE WHEN: you want a flat list of URLs on ONE domain, fast. NOT FOR: web-wide search (use `search`); URLs filtered by topic with an LLM (use `scan` with `criteria`); recursive content crawl (use `site_start`). EXAMPLE: 'list all URLs on docs.example.com'. Sync, ~2-15s on 'primary' (sitemap+robots+RSS), ~30-60s on 'extended' (also Wayback+Common Crawl). `query` enables BM25 ranking; `score_threshold` filters by relevance score.

Scan

USE WHEN: you want URLs on ONE domain, optionally filtered by an LLM `criteria` ('product pages only', 'blog posts about AI'). Like `map` but with AI-driven filtering and a generated SiteScanConfig. NOT FOR: web search (use `search`); just dumping every URL (use `map`); fetching content (use `site_start` with `scan_only=false`). EXAMPLE: 'find pricing pages on stripe.com' → criteria='pricing pages'. Sync, ~3-15s. `criteria` is plain English. `query` adds BM25 ranking. Returns `urls[]` plus `gen

Site Start

USE WHEN: you want to RECURSIVELY crawl pages from a seed URL — best-first traversal across the worker pool, with content extraction. Returns {job_id} — poll `site_status`. NOT FOR: just listing URLs (use `map` or `scan`); a single page (use `scrape`); structured data extraction (use `extract_start`). EXAMPLE: 'crawl docs.example.com, max 50 pages, depth 3'. Async, 10s-5min depending on `max_urls`. `mode='traverse'` (default) fetches content; `mode='map'` only enumerates URLs. `query` enables b

Site Status

USE WHEN: polling a `site_start` job by its job_id. Returns {status, progress, urls_discovered, urls_crawled, crawl_job_id?, results?, error?}. Per-URL results land at /v1/crawl/jobs/{crawl_job_id}/result/{idx} once content fetch starts.

Extract Start

USE WHEN: you want STRUCTURED DATA (JSON) from one or more known URLs. Returns {job_id} — poll `extract_status`. NOT FOR: just markdown (use `scrape`); recursive crawl + extract (use `site_start` then `extract_start` on results); finding URLs first (use `enrich_start` for plan→find→extract). EXAMPLE: 'get product name + price from https://shop.com/p/42, plus these 5 follower URLs' → query='product name and price', extra_urls=[...]. `method='auto'` (default) lets the system pick LLM vs CSS-schem

Extract Status

USE WHEN: polling an `extract_start` job by its job_id. Returns {status, progress, results?, download_url?, error?}. `results[]` is one entry per URL with the extracted JSON. `download_url` is a presigned link to the full result blob.

Enrich Start

USE WHEN: you want a RESEARCH-GRADE TABLE built from a fuzzy query — the system plans entities/criteria/features, finds candidate URLs via search, extracts per-row data. Returns {job_id} — poll `enrich_status`. NOT FOR: known URLs (use `extract_start`); a single quick answer (use `search_synth_start`); just URL discovery (use `scan`). EXAMPLE: 'top 10 AI code review tools with pricing and integrations' → enrich plans columns, searches the web, extracts per-tool, returns a table. Long-running, 3

Enrich Status

USE WHEN: polling an `enrich_start` job by its job_id. Returns {status, phase: plan|resolve|extract|complete, progress, rows?, error?}. Job may pause at `plan` or `resolve_urls` waiting for confirmation if `auto_confirm_*` was disabled — call /v1/enrich/jobs/{id}/continue to advance.

Configure

USE WHEN: you want to translate a plain-English description into a ready-to-POST request body for one of the /v1 endpoints. The Phase 4 super-generator. Optionally returns curl/python/node/go snippets. NOT FOR: actually running the operation — `configure` only returns the body shape; you still need to call the matching tool. EXAMPLE: 'I want to scrape every product page on shop.com and get name, price, image' → returns the right tool + body. Sync, ~2-8s. `service` hints which endpoint family to