Skip to content

feat(page-cluster): self-tuning recursive clustering with data-driven landmark scoping#916

Merged
YusukeHirao merged 5 commits into
devfrom
feat/page-cluster-self-tuning-v2
Jul 13, 2026
Merged

feat(page-cluster): self-tuning recursive clustering with data-driven landmark scoping#916
YusukeHirao merged 5 commits into
devfrom
feat/page-cluster-self-tuning-v2

Conversation

@YusukeHirao

@YusukeHirao YusukeHirao commented Jul 13, 2026

Copy link
Copy Markdown
Member

Summary

@d-zero/page-cluster に自己チューニング再帰クラスタリングを実装し、landmark 抽出のハードコード depth 制約を撤廃、データ駆動で site-wide / section-local chrome を自己発見する仕組みに置き換える。

4 コミットで構成

  1. first-party stylesheet 判定の同ホスト解決 — blocking key の生成でページ自身のホスト情報を使用し、同一ホスト判定を正確化
  2. 自己チューニング再帰クラスタリング (Stage A + Stage B) — Stage A はブロック内 auto-cut + containment assignment、Stage B は cross-block 統合（quorum core + shape Jaccard + L2 signature + shell corroboration）
  3. landmark 抽出を form / search 対応に拡張 — HTML5 sectioning element + ARIA landmark role の 6 種（header / footer / nav / aside / form / search）を対象化
  4. depth 制約撤廃 + コーパスレベル chrome 自己発見 — extractLandmarks は全 landmark instance を配列で返す（BREAKING）、shellQuorum は per-token 頻度への autoCutThreshold 適用、コーパス頻度分布に auto-cut を適用して local chrome を判定・Stage A へトークン再注入

BREAKING CHANGES

  • ExtractLandmarksResult のフィールド型が string | undefined → string[]
  • 各 landmark type について「浅いもの勝ち」の 1 マッチではなく document 順の全マッチが返される
  • 下流の chrome / content 判別は corpus / unit 頻度分析に完全移譲

実装機構の要点

  • 同一 autoCutThreshold プリミティブを 3 階層で再帰使用: (a) Stage A のマージ高さカット (b) Stage B unit 内 shell quorum (c) コーパスレベル landmark signature 頻度
  • 既存 80% 定数は退化ケース (signature 数 <2、全同頻度) 専用のフォールバック clamp に降格
  • 局所 chrome (auto-cut 以下 & count ≥ 2) の tokens を該当ページの block token set に再注入することで、Stage A の Jaccard-based clustering が構造的差異を検出可能に

Test plan

  • yarn build / yarn lint / yarn test (1,554 tests pass, cspell clean)
  • 新規 spec 4 個 (find-shallowest-elements, per-page-landmark-signatures, extract-landmarks 再構成, resolve-page-cluster-keys コーパスレベル chrome 判別 3 挙動)
  • 既存 spec の更新 (breaking change の期待値書き換え)
  • 4 コーパスでの実データ検証: 2 コーパスは前計画のゲート値近傍、2 ホールドアウトコーパスは template 家族での意味論的分離を確認

🤖 Generated with Claude Code

@YusukeHirao
fix(page-cluster): resolve first-party stylesheet ties via page host
542c533 
filterFirstPartyStylesheetHrefs picked a single "dominant host" by counting
how many pages reference each stylesheet host, since it never had access to
each page's own URL to compare against directly. When a third party (e.g. a
sitewide Google Fonts request) is referenced by exactly as many pages as the
real first-party host, the tie-break silently keeps whichever host happened
to be counted first — confirmed on a real 302-page crawl where every single
page loaded both, and the third party won.

Adds an optional `host` field to PageClusterSignals: when a caller supplies
it, each page's stylesheetHrefs are filtered by direct comparison against
that host instead of the batch-wide vote, eliminating the tie case entirely.
Omitting it keeps the existing dominant-host fallback unchanged.
@YusukeHirao
feat(page-cluster): add self-tuning recursive clustering
4cb5e83 
Stage A: auto-cut threshold with clamping + containment assignment
inside each block. Stage B: cross-block recursive merge using
quorum-80% cores, shape-Jaccard ≥ 0.9, L2 multi-set signatures
with shell corroboration, iterating until fixed point (MAX_ROUNDS=10).

Also promotes autoCapMainDepth default to true.

BREAKING CHANGE: cluster keys may differ from previous versions
@YusukeHirao
feat(page-cluster): extend landmark extraction to form and search
ec182f8 
Adds form (role="form") and search (<search> tag / role="search")
landmark types to extractLandmarks. Bare <form> without a role is
intentionally excluded — it has no implicit landmark role under
HTML-AAM and is often page-specific content.

Also widens the shell-corroboration signal in mergeCrossBlockClusters
to include aside, form, and search in addition to header/nav/footer.
@YusukeHirao
feat(page-cluster): drop depth heuristic, discover local chrome via a…
ae5dc24 
…uto-cut

Landmark extraction now collects every genuinely-closed instance per type
(no shallowest-wins depth rule); ExtractLandmarksResult switches to
string[] arrays and downstream consumers pick their own canonical instance
via corpus-signature frequency.

shellQuorum runs autoCutThreshold over the per-landmark-token page-
frequency histogram (was fixed 80% quorum). Corpus-level chrome discovery
in resolvePageClusterKeys applies the same primitive to landmark-instance
signatures, and reinjects tokens of local (below-cut, non-singleton)
signatures into Stage A block token sets so section-local template
variants become structurally visible.

BREAKING CHANGE: ExtractLandmarksResult fields change from string|undefined
to string[]; every landmark instance is now returned in document order
instead of just the shallowest match per type.
@YusukeHirao YusukeHirao requested a review from yusasa16 as a code owner July 13, 2026 01:58
@YusukeHirao YusukeHirao merged commit b239c9a into dev Jul 13, 2026
6 checks passed
@YusukeHirao YusukeHirao deleted the feat/page-cluster-self-tuning-v2 branch July 13, 2026 03:48
@YusukeHirao YusukeHirao mentioned this pull request Jul 14, 2026
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yusasa16 yusasa16 Awaiting requested review from yusasa16 yusasa16 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@YusukeHirao