Apify

Apify

Web scraping and automation platform. Discover scrapers in the Store, run them with custom input, monitor execution, and retrieve structured results.

Entity Hierarchy

• Actors (scrapers/automations) are the core unit. Each Actor has a unique id (e.g. apify/web-scraper) and versioned Builds.
• Runs are execution instances of an Actor. A run has a status (READY, RUNNING, SUCCEEDED, FAILED, TIMED-OUT, ABORTED) and a defaultDatasetId.
• Datasets hold tabular output from runs. Each dataset has paginated Items (the actual scraped data rows).
• Key-Value Stores hold arbitrary key→value pairs (JSON, HTML, screenshots). Each run gets a default store. The INPUT key holds the run's input config.
• Request Queues manage URL frontier for crawling Actors. Items are URLs with metadata.
• Schedules trigger Actor runs on a cron schedule.
• Webhooks notify external systems on run state changes.

Capability areas

Store & discovery

Keyword search across the public Actor marketplace yields ids, pricing hints, and popularity—use that before committing to a scraper.

Your Actors vs Store Actors

Owned Actors live in a separate inventory from marketplace listings; open a catalog entry to read README, input schema, and recent run stats.

Execution lifecycle

Runs are started with structured JSON input and return immediately while work continues in the background. Historically completed runs, live status, cancel, resurrect, and advanced “morph” controls all hang off the run object.

Outputs

Tabular datasets expose paginated rows; key-value stores hold blobs like screenshots or HTML snapshots; request queues show crawler frontiers. Each finished run points at its default dataset and default store ids.

Account

Profile reads expose quota consumption and plan ceilings.

Workflows

1. Find and run a scraper: Marketplace search → inspect Actor metadata and input schema → launch with validated JSON → poll run state until success → pull dataset pages for the result grid.
2. Monitor active runs: Filter the global run list, open a single run for diagnostics, cancel if it is wedged.
3. Access run artifacts: Read the run payload for defaultDatasetId vs defaultKeyValueStoreId, then fetch tabular rows or individual store records (e.g. rendered PNGs).

Gotchas

• Async execution: Starts return immediately—poll run state until terminal status. Duration ranges from seconds to hours.
• Actor ID format: Marketplace Actors use vendor/slug style ids; first-party builds often use opaque hashes.
• Dataset pagination: Default page sizes are modest—walk with offset/limit when exports spill past the first window.
• Token scope: API tokens are full-account; assume least-privilege operationally even though the vendor does not offer sub-roles.
• Run memory: Memory/time drive compute units—compare remaining allowance on the account snapshot when planning big scrapes.