Providers
Providers
Web archives and threat intel indexes behind one record shape. Pick a source to see its endpoint and its limits and the traps.
Every source answers the same call, discover(domain, options), and returns the same record. What differs is where it gets its URLs, whether it knows when it saw them, and how much it's willing to hand over in one go.
The archives with a CDX index (Wayback, Arquivo.pt, Vefsafn, Common Crawl) know capture dates, so from and to work on them. The security indexes (AlienVault OTX, URLScan, VirusTotal) know URLs people and scanners ran into, sometimes ones no archive bothered to keep.
| Format | What it knows | ||
|---|---|---|---|
| AlienVault OTX | JSON pages | Threat intel URL lists. Keyless, but the public endpoint gets grumpy under load. | no key |
| Arquivo.pt | CDX NDJSON | The Portuguese web archive. CDX with dates, and a hard cap per request. | no key |
| Common Crawl | CDX text | Crawl indexes, one per year for the last five. A broken index skips, the rest answer. | no key |
| URLScan | JSON search pages | Pages people scanned. Answers without a key at small volumes, more with one. | key optional |
| Vefsafn | CDX NDJSON | The Icelandic web archive. Ignores limit and sends everything, the collector stops it. | no key |
| VirusTotal | JSON v3 pages | URLs VirusTotal has seen for the domain. No key, no answer. | key required |
| Wayback Machine | CDX text | The Internet Archive's CDX index. One streamed answer with dates, subdomains included. | no key |
Which one first?
For breadth on an old domain, Wayback. For what's been poked at recently, URLScan and AlienVault OTX. Not sure? provider: "all" asks everyone and shows who answered, which beats guessing.