feat(page-cluster)!: trim options, add CLI and onProgress#918
Merged
Conversation
… API BREAKING CHANGE: the `mergeRareLandmarkClusters`, `landmarkRarityThreshold`, `landmarkGateSimilarityThreshold`, and `autoCapMainDepth` options are removed from `resolvePageClusterKeys`. The `autoCapMainDepth`-driven <main> depth cap is now unconditional and matches the previous default (`true`); the rare-landmark merge feature was never validated on real crawl data and is dropped along with its `mergeLandmarkAffinedClusters` implementation. BREAKING CHANGE: `package.json` `exports` shrinks from 17 subpaths to 4 (`.` / `./extract-landmarks` / `./resolve-landmark-variant-keys` / `./resolve-page-cluster-keys`). Internal helpers stay in `src/` for in-package imports; downstream callers relying on removed subpaths must build against the public surface. New: - `bin: page-cluster` CLI. JSONL in, JSONL out, single `--content-block- attribute` flag, `--help` / `--version`, stderr progress lines. Uses Node standard APIs only. - `resolvePageClusterKeys` `onProgress` callback + `ProgressEvent` discriminated union covering pass0 / pass1 / pass1b / stage-b start. - `derivePathClusterKeys` — auto-cut-driven URL-path depth selector. Exported as a building block and reachable via `resolveBlockingGroupKeys`'s new `pathDepth: 'auto'`; `resolvePage- ClusterKeys` continues to use static depth 1 for now (opt-in staging, same as previous auto-cut adoptions). - `@example` blocks on the 4 public APIs. - Rewritten README aimed at external users, CLI-first. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
前 PR #917（streaming API）に続く実用化整備。オプション削減、公式 CLI、進捗コールバック、公開 API 表面の縮小を一括で入れる。
BREAKING CHANGES
resolvePageClusterKeysから 3 オプションを撤去:mergeRareLandmarkClusters/landmarkRarityThreshold/landmarkGateSimilarityThreshold— 実データ未検証のフラグ。関連するmergeLandmarkAffinedClusters実装ごと削除autoCapMainDepth— 常時 ON に固定。既存デフォルトtrueと挙動同一package.jsonのexportsを 17 → 4 サブパスに削減:.(tokenize) /./extract-landmarks/./resolve-landmark-variant-keys/./resolve-page-cluster-keyssrc/に残置。パッケージ内相互 import は継続追加
bin: page-cluster。JSONL 入出力、--content-block-attributeフラグ、--help/--version、stderr 進捗ログ。Node 標準 API のみ使用onProgressコールバック:resolvePageClusterKeysのオプションに追加。ProgressEvent判別共用体で pass0 / pass1 / pass1b / stage-b 開始をイベント化derivePathClusterKeys: URL パス深さの auto-cut ベース自己決定関数。resolveBlockingGroupKeysにpathDepth: 'auto'として opt-in。resolvePageClusterKeysは現時点で有効化しない（既存回帰保証、実データ検証は後続 PR）@example: 4 公開 API 全てに runnable example を付与検証
yarn build/yarn lint/yarn test（759 spec）all pass意図的に本 PR に含めない
derivePathClusterKeysの main 経路への配線（実データ検証を要するので別 PR）onProgressイベント（現在は開始 1 回のみ発火。将来拡張）Test plan
yarn build/yarn lint/yarn testall pass🤖 Generated with Claude Code