Skip to content

AST 'uses' relation cross-products every local class with every imported symbol in a file (Python), false positives backfilled to confidence_score=0.5 #2236

Description

@alisalehi02

Problem

The AST uses relation for Python (extractors/resolution.py, the from .X import A, B, C resolution pass) does not check whether a local class actually references an imported symbol. It cross-products every class defined in a file against every name imported into that file, emitting a uses edge for each pair unconditionally:

# extractors/resolution.py, ~line 1922
local_classes = [
    n["id"] for n in file_result.get("nodes", [])
    if n.get("source_file") == str_path
    and not n["label"].endswith((")", ".py"))
    and n["id"] != _make_id(stem)
    and n.get("file_type") != "rationale"
]
...
# ~line 1988
for name in imported_names:
    tgt_nid = stem_to_entities[target_fq].get(name)
    if tgt_nid:
        for src_class_nid in local_classes:
            new_edges.append({
                "source": src_class_nid,
                "target": tgt_nid,
                "relation": "uses",
                "confidence": "INFERRED",
                "source_file": str_path,
                "source_location": f"L{line}",
                "weight": 0.8,
            })

There is no check that src_class_nid's body (fields, annotations, method bodies) actually mentions the imported name — every class in the file gets an edge to every import in the file, module-wide co-occurrence standing in for a real reference.

Compounding this, these edges never set confidence_score (only weight: 0.8). At export time, export.py:

# export.py:159
_CONFIDENCE_SCORE_DEFAULTS = {"EXTRACTED": 1.0, "INFERRED": 0.5, "AMBIGUOUS": 0.2}
...
# export.py:302-304
if "confidence_score" not in link:
    ...
    link["confidence_score"] = _CONFIDENCE_SCORE_DEFAULTS.get(conf, 1.0)

backfills the missing field to a flat 0.5 for every one of these edges. So every false-positive uses edge is also indistinguishable from a genuinely graded low-confidence edge — it shows up as a hard cliff at exactly confidence_score: 0.5, relation: uses, confidence: INFERRED.

Reproducer

Real-world repro from a FastAPI backend (backend/routers/admin.py):

class AgentConfigBody(BaseModel):
    agent_name: str
    task_type: Optional[str] = None
    model_name: str
    temperature: Optional[float] = None
    is_active: bool = True

AgentConfigBody has zero fields typed as AgentConfig, Subscription, or Payment. Yet the extracted graph contains:

AgentConfigBody --uses--> AgentConfig   [INFERRED 0.5]
AgentConfigBody --uses--> Subscription  [INFERRED 0.5]
AgentConfigBody --uses--> Payment       [INFERRED 0.5]

...because admin.py happens to import all three ORM models somewhere else in the file for unrelated route handlers. The same pattern repeats for every other Pydantic body class in that file (QuotaOverrideBody, etc.) and in routers/plans.py (PackBody, PackUpdate, PlanUpdate each "using" Plan, Subscription, Payment).

On this ~2000-node graph (241 files, mixed FastAPI backend + React frontend), 191 of 5176 edges (3.7%) carried this exact signature: relation=uses (187) or relation=indirect_call (4, same missing-confidence_score pattern, worth checking separately), confidence=INFERRED, confidence_score=0.5 exactly. Filtering these out and re-clustering changed community structure — communities that looked artificially merged (e.g. "Admin API Routes" and "Billing Models") cleanly separated once the phantom edges were removed.

Why this is a different bug from #540

#540 tracks the LLM subagents' semantic extraction defaulting to 0.5 for genuinely-uncertain INFERRED edges — a calibration/prompting problem, already mitigated by the current extraction-spec's forced discrete rubric (0.55/0.65/0.75/0.85/0.95, "never 0.5").

This issue is about the deterministic AST pass (no LLM involved) producing edges that are structurally guaranteed to be false positives whenever a file defines >1 local class and imports >1 external symbol — it's a correctness bug in the resolver, not a calibration problem. The flat 0.5 is a downstream symptom (the missing-field default in export.py), not the root cause.

Suggested fix

  1. In resolution.py's import-resolution pass, before emitting a uses edge for (src_class_nid, tgt_nid), check whether src_class_nid's own node span (or at minimum its field-annotation text) actually contains the imported name as a token. Skip the edge if not — this turns the cross-product into a real (if approximate) reference check.
  2. Independently, export.py's _CONFIDENCE_SCORE_DEFAULTS["INFERRED"] = 0.5 silently masks any future producer that forgets to set confidence_score behind a value that looks like a real score rather than "unset." Consider either requiring confidence_score at the point of edge creation (fail loud in dev/tests) or writing an out-of-band sentinel (e.g. null, or a confidence_score_source: "default" flag) so downstream consumers (and this exact investigation) don't have to reverse-engineer which edges were actually scored vs. defaulted.

Environment

  • graphifyy 0.9.28 (pip show graphifyy)
  • Repro corpus: 241 files (223 code, 18 docs), FastAPI + SQLAlchemy backend, React frontend
  • Found via /graphify skill's Step 4.5 health check + manual confidence_score distribution audit

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions