Internet Archive

Internet Archive

Two distinct search surfaces plus Wayback web archaeology. No auth required.

Which surface to use

Catalog metadata matches titles/subjects/collections — not words on page 282. Full-text search matches OCR inside volumes. Wayback matches live-web URLs over time. Do not substitute one for another.

Wayback workflows

• Snapshot presence — Ask whether a URL has any capture, returning the nearest timestamp plus the canonical archive link. Treat empty availability answers skeptically for large sites; CDX queries are the authoritative existence signal.
• Batch presence — Send many {url, tag?, timestamp?, closest?} tuples in one round trip and read per-URL outcomes grouped by optional tags.
• CDX discovery — Mine the index for metadata rows (urlkey, timestamp, mime type, HTTP status, digest, byte length). Tune match scope (exact URL, prefix, host, domain), YYYYMMDD windows, field projections, filters like statuscode:200, and JSON/CSV/text encodings.
• Timelines — Collapse up to ~100 chronological captures for a URL, each with status, mime, digest, and replay link; backed by the same CDX machinery for stability.
• Content replay — Pull stored markup for a capture. Supply YYYYMMDDHHMMSS for a precise revision or omit to follow the closest mirror. raw strips the Wayback chrome; view keeps the standard wrapper.

Catalog and full-text workflows

• Metadata discovery — Query with Lucene syntax: title:(plato) AND mediatype:texts, collection:opensource, creator:(darwin), identifier:some-item-id. Paginate with page/rows (max 100 rows).
• Large harvests — Use scrape with cursor; upstream requires count ≥ 100 per batch.
• In-book phrases — Full-text search returns highlight snippets and page_num when indexed. Scope to one item by including identifier:… in the query.
• Item drill-down — After search, fetch metadata + file list for the identifier, then open details_url on archive.org for human viewing.

Timestamp formats (Wayback)

• Availability API: ISO 8601 (YYYY, YYYY-MM, YYYY-MM-DD, YYYY-MM-DDTHH:mm:SSz)
• CDX search: YYYYMMDD for date range filters (from_date, to_date)
• Content retrieval: YYYYMMDDHHMMSS for specific snapshots
• CDX results: Timestamps returned as YYYYMMDDHHMMSS strings

Gotchas

• Rate limits: IA throttles per egress IP (~60 CDX calls/min). MCPBundles routes through Cloudflare when AWS is throttled and fails over between egress paths on 429 instead of blocking for a minute. Be respectful; use batch availability for multiple URLs. Keep CDX search limits reasonable (default 100).
• Replay URLs: Each CDX/timemap row includes replay_url. For JPEG/MP4 and other media, Wayback uses the im_ modifier (/web/{ts}im_/…). For HTML without chrome use id_. Default /web/{ts}/… is the viewer page — fine for pages, wrong for downloading a camera still.
• CDX match scope: Use prefix on the full asset URL (e.g. https://…/jamcams.tfl.gov.uk/00001.07450.jpg), not domain on a hostname alone — domain expands to *.tfl.gov.uk/* and misses S3-hosted paths. Bare domains (example.com) are normalized to https://example.com for host/domain matching.
• CDX API is slow: Broad prefix searches can take 30-45 seconds. Narrow with filter (e.g. statuscode:200 mimetype:image/jpeg), date ranges, or lower limit.
• String parameters: Timestamps and dates must be strings, not integers. Quote them when calling from CLI.
• Flaky availability: The availability endpoint returns empty for some well-known URLs. CDX search is more reliable for confirming archive existence.
• Sparse history: Wayback captures opportunistically (when crawled), not on the source feed's refresh cadence. A live traffic camera that updates every 3 minutes may only have dozens of archived stills per year — not a continuous DVR.
• Search inside one book can be slow or time out; full-text search across the corpus is usually faster for discovery.
• Open Library (separate bundle) is bibliographic metadata — not Internet Archive OCR full-text or Wayback.