feat(crawler): pause dealer on network outage and resume in-place

[YusukeHirao]

Background

Local network handoff (WiFi → tether / VPN flip / ISP DNS hiccup) mid-crawl can
exhaust each in-flight worker's retry budget within ~30 seconds even after the
network recovers seconds later. Worse, during longer outages (5-30 minutes) the
dealer keeps pulling new URLs from the queue and each spends 6-30s spinning
through retries before being recorded as status = -1 failure. For a 70k-URL
crawl a 10-minute outage can burn ~10 hours of "retry waste" wall-clock and
produce tens of thousands of false-failure rows.

The recent success-guard fix (in-progress PR) prevents the cascade
dnsBurnedHostCache poisoning but does NOT address retry waste or
false-failure recording during the outage window. This issue tracks the next
layer: actually pause work until the network is back.

Proposed design

Detection

• Maintain a sliding window of recent DNS / network errors across all hosts
  inside Crawler.
• When N (e.g. 5) DNS errors occur within W (e.g. 10) seconds across at least
  M (e.g. 2) distinct hosts → declare "network outage suspect".
• On suspect-mode entry, run an active dns.lookup probe against a stable
  host (e.g. one that already succeeded in this session — see the
  #successfulHosts set landed by the success-guard PR).
• If the probe succeeds: false alarm, exit suspect mode immediately.
• If the probe fails: confirmed outage, close the gate.

Gate / pause

• Crawler exposes an internal await this.#networkGate.wait() that worker
  callbacks (the dealer's per-URL task body) call BEFORE doing any network
  work.
• When the gate is open (default), wait() resolves immediately.
• When the gate is closed, wait() blocks until the gate opens.
• Achieves "dealer callback awaits" semantics without modifying
  @d-zero/dealer internals — the dealer just sees a long-running task.

Recovery

• While the gate is closed, a probe loop runs every K seconds (e.g. 10s)
  calling dns.lookup on the stable host.
• On first successful probe: open the gate. All awaiting workers wake up and
  proceed.
• Gate opens are level-triggered (no race conditions with concurrent awaiters).

In-flight handling

• URLs already past the gate and inside fetchDestination continue as-is.
  Their retry mechanism handles transient failures. If they exhaust retries
  during the outage they fail with status = -1 — the SAME behavior as today
  for genuine failures. Acceptable, since --retry-failed can recover them.
• To minimize even this loss, wrap fetchDestination with a check: if the
  gate closes while a request is in flight, abort that request and re-queue
  the URL to the front of linkList (linkList.unshift equivalent) so it is
  the first to resume when the gate opens.

Implementation notes

• Pure JS, no native bindings (dns.lookup is sufficient for probing).
• Probe host selection: prefer a host that succeeded in this session over a
  hardcoded 1.1.1.1 (avoids depending on external infrastructure for our
  own health check).
• Configuration knobs (window size, threshold, probe interval) live on
  CrawlerOptions with sensible defaults but can be tuned per crawl.
• Verbose logging: [network] outage suspected — pausing N workers,
  [network] recovered — resuming so operators can see what happened.

Alternatives considered

• OS-level network change events via child_process.spawn('route -n monitor')
  or native bindings — rejected for portability and dependency cost. Polling
  detection is sufficient for our retry-waste prevention.
• Libraries: is-online, internet-available — same active-probe approach
  as proposed here but pull in dependencies. The implementation is small
  enough to inline.

Out of scope

• Predicted-discard placeholder leak (crawler.ts:980 missing emit('skip')).
• Inventory mode persistence on resume / retry-failed.
• Long-term observability of network instability across sessions.