Custom providers

Found another passive index? It takes one class and one call.

import {
  create,
  Provider,
  register,
  resolveDomain,
  UrlCollector,
  type DiscoveredUrl,
  type DiscoverOptions,
} from "@agntn/urls";

class MyIndex extends Provider {
  static readonly key = "myindex";

  get capabilities() {
    return { discover: true };
  }

  async discover(domain: string, options?: DiscoverOptions): Promise<DiscoveredUrl[]> {
    const target = resolveDomain(domain, this.name);
    const collector = new UrlCollector(options, domain);
    const api = `https://index.example/urls?domain=${target}`;
    const rows = await this.getJSON<{ url: string; seen?: string }[]>(api, { signal: options?.signal });

    for (const row of rows) {
      if (collector.done) break;
      collector.push(this.name, row.url, api, row.seen);
    }
    return collector.results;
  }
}

register(MyIndex, {
  key: "myindex",
  capabilities: { discover: true },
  load: () => Promise.resolve(MyIndex),
});

const mine = await create("myindex");

Three things not to skip

• Build the request from resolveDomain(). It turns a domain or a full URL into a bare hostname and rejects anything that could sneak a path into your query. Never paste the raw input into a URL.
• Push through UrlCollector. It applies every filter, deduplicates, and tells you when the limit is reached through done. Check it between pages and stop asking.
• Forward options.signal. getJSON and getTextLines on Provider take it and add the timeout from create().

getTextLines streams a text answer line by line and cancels the rest once you break, which is what you want for a CDX dump. Wayback shows the whole pattern.

After register()

The key works with create(), discoverAll(), the automatic pick and the CLI's -p, as long as the process that runs them registered it first. Registering the same key again replaces the earlier entry.

A source that belongs in the package itself goes into the built-in manifest instead, src/providers/index.ts, with its own module so it loads lazily. Pull requests welcome.