Skip to content

feat(page-cluster)!: streaming API for large corpora#917

Merged
YusukeHirao merged 2 commits into
devfrom
feat/page-cluster-streaming-scale
Jul 14, 2026
Merged

feat(page-cluster)!: streaming API for large corpora#917
YusukeHirao merged 2 commits into
devfrom
feat/page-cluster-streaming-scale

Conversation

@YusukeHirao

@YusukeHirao YusukeHirao commented Jul 14, 2026

Copy link
Copy Markdown
Member

Summary

resolvePageClusterKeys を async ファクトリ入力に変え、コーパスがマシンメモリを超えても完走できるようストリーミング経路を追加。従来の array 入力 API は syntactic sugar resolvePageClusterKeysFromArray として保持。

BREAKING CHANGE

// Before
resolvePageClusterKeys(pages: PageClusterSignals[], options?): string[]

// After
resolvePageClusterKeys(
    pages: () => Iterable<PageClusterSignals> | AsyncIterable<PageClusterSignals>,
    options?,
): Promise<string[]>

// Convenience for callers with an in-memory array
resolvePageClusterKeysFromArray(pages: PageClusterSignals[], options?): Promise<string[]>

動機

前 PR (#916) までの実装は全ページを配列に materialize する設計で、10 GB / 176k ページ級の実データで OS SIGKILL / V8 heap OOM が確定。実務の大規模クロールに耐えない。

変更点

経路分岐

コーパスサイズ 経路 挙動
≤ CORPUS_INLINE_THRESHOLD (20,000) in-memory 前 PR と byte-identical
> 20,000 streaming 3 パス構成

streaming 経路の 3 パス:

  1. Pass 0 — HTML-free blocking: paths / stylesheetHrefs / host のみ全ページ走査。resolveBlockKeys (新規、既存の blocking / orphan-reassignment / first-party フィルタを合成) で全 blockKey を確定
  2. Pass 1 — per-block reservoir sampling: 各ブロックから最大 BLOCK_SAMPLE_SIZE (100) ページを Algorithm R でサンプル。サンプルに Stage A を実行し、per-block maxMainDepth / local-chrome signatures / per-cluster member token sets を学習物として保持
  3. Pass 1b — Jaccard 割当: 非サンプルページを再ストリームし、そのブロックの学習物と同一の変換空間（chrome 再注入、maxMainDepth 適用、tokenize）でトークン化 → 最良 Jaccard クラスタに割当

Stage B (mergeCrossBlockClusters) はサンプルベースの CrossBlockUnit 群に対して従来通り実行。

メモリ削減

  • CrossBlockUnit.memberLandmarks: ExtractLandmarksResult[] → memberLandmarkInstances: PerPageLandmarkInstance[][]。事前トークン化して保持することで per-page ~10× のメモリ削減。Stage B 側 (shellQuorum) も再トークン化を停止
  • mergeCrossBlockClusters に opt-in capMembers を追加。merged group を各 merge 後にダウンサンプリング。streaming 経路のみ有効化、in-memory 経路には影響なし

決定論

  • reservoir sampling は Mulberry32 PRNG を block key の FNV-1a hash でシード。同じコーパス・同じ入力順で同じ結果

検証

既存 4 コーパス回帰（in-memory 経路、ゲート値完全一致）

ページ数 クラスタ数 resolve 時間 peak RSS
302 9 2.4 s 251 MB
8,936 21 106 s 1.2 GB
1,416 63 77 s 3.2 GB
89 3 0.7 s 115 MB

大規模実データ（streaming 経路）

  • 176,046 ページ / 10 GB アーカイブ
  • 完走時間: 27.7 分
  • peak RSS: 1.4 GB、peak heap: 2.8 GB（従来 8 GB heap でも OOM）
  • クラスタ数: 232

テスト

  • 767 spec 全 pass
  • 新規 spec: pass0-blocking.spec.ts、reservoir-sample.spec.ts、resolve-page-cluster-keys-streaming.spec.ts

意図的に本 PR に含めていないもの（別 PR で継続）

  • 公式 CLI（JSONL パイプ、bin フィールド）
  • options 縮小と自己解決化（landmarkGateSimilarityThreshold 撤去等）
  • package.json の exports を 4 サブパスに絞る作業
  • README 刷新（Quickstart + アルゴリズム walkthrough）
  • 進捗コールバック (onProgress)

Test plan

  • yarn build / yarn lint / yarn test all pass
  • 既存 4 コーパスで前 PR ゲート値 (9 / 21 / 63 / 3) を維持
  • 176k ページの実データで完走確認（1.4 GB RSS / 27.7 min）

🤖 Generated with Claude Code

@YusukeHirao @claude
feat(page-cluster)!: add streaming API for corpora that overflow memory
d54c8e1 
BREAKING CHANGE: `resolvePageClusterKeys` is now async and takes a factory
function `() => Iterable<Page> | AsyncIterable<Page>`. Callers with a
materialized array can use the new `resolvePageClusterKeysFromArray`
convenience wrapper; existing sync array-based use cases keep working with a
one-line change.

Corpora at or below `CORPUS_INLINE_THRESHOLD` (20,000 pages) still take the
byte-identical in-memory path; larger corpora are processed via three passes:

- Pass 0 (HTML-free) — read blocking signals only and compute block keys
  with the existing blocking / orphan-reassignment / first-party filter
  pipeline.
- Pass 1 (per-block reservoir sampling) — draw up to `BLOCK_SAMPLE_SIZE`
  (100) pages per block, run Stage A on the sample, emit its CrossBlockUnits,
  and save learned artifacts (per-block `maxMainDepth`, local-chrome
  signatures, per-cluster member token sets) for the assignment step.
- Pass 1b (streaming assignment) — Jaccard-assign every non-sample page to
  its block's nearest sample-derived cluster.

Additional memory reductions:

- `CrossBlockUnit.memberLandmarks: ExtractLandmarksResult[]` becomes
  `memberLandmarkInstances: PerPageLandmarkInstance[][]`. Landmarks are
  pre-tokenized at unit creation instead of being re-tokenized by shellQuorum
  each round.
- `mergeCrossBlockClusters` accepts an opt-in `capMembers` option that
  down-samples merged groups after each merge. Streaming callers pass 100;
  the in-memory path omits it to keep validated corpora unchanged.

Small-corpus regression preserved on the four historically validated
corpora (302 / 8,936 / 1,416 / 89 pages → 9 / 21 / 63 / 3 clusters). New
streaming path validated on a 176,046-page real crawl: completes in ~28 min
at 1.4 GB peak RSS / 2.8 GB peak heap, well within an 8 GB budget.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@YusukeHirao YusukeHirao requested a review from yusasa16 as a code owner July 14, 2026 02:02
@YusukeHirao YusukeHirao merged commit d360d11 into dev Jul 14, 2026
6 checks passed
@YusukeHirao YusukeHirao deleted the feat/page-cluster-streaming-scale branch July 14, 2026 02:10
@YusukeHirao YusukeHirao mentioned this pull request Jul 14, 2026
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yusasa16 yusasa16 Awaiting requested review from yusasa16 yusasa16 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@YusukeHirao