Providers
Wayback Machine
The CDX index of the Internet Archive. No key. One streamed text request with dates and subdomains included.
Source
Wayback Machine
- provider: "wayback"
- web.archive.org
The Internet Archive's CDX index. One streamed answer with dates, subdomains included.
- Format
- CDX text
- Paging
- one streamed request
- Dates
- firstSeen · lastSeen
- Key
- none
Access
- Load
await create("wayback") - CLI
urls discover example.com -p wayback - Key
nothing, no key and no account - Endpoint
https://web.archive.org/cdx/search/cdx
What it knows
Everything the Internet Archive captured under the domain since 1996. For an old domain that's the deepest source by far, and every row has a timestamp.
How it asks
One request to /cdx/search/cdx with matchType=domain, output=txt and fl=original,timestamp. The answer streams line by line and the reader is cancelled the moment the collector has enough.
Traps
url=domain/*only matches the apex host,www.included. Subdomains never came back that way, so the provider asks withmatchType=domain, like urlfinder's*.domain/*.collapse=urlkeywould dedupe on the server, but that variant times out from some networks while the plain query answers at once. The collector dedupes instead.- Rows are captures, not URLs.
http://example.com:80/captured in 2002 andhttp://example.com/captured in 2020 are one URL after dedup. The first spelling wins and the dates widen. - On a big domain the full dump is huge. Set a
limitor a filter, or run your own CDX server and pointbaseUrlat it.