Common Lisp support, rebased onto v8 and moved into extractors/ - #2274
Common Lisp support, rebased onto v8 and moved into extractors/#2274fade wants to merge 11 commits into
Conversation
Add an imperative tree-sitter-commonlisp walker that extracts packages, classes, functions, methods, macros, generic-function dispatch, and calls from Common Lisp source. Registered alongside the other custom extractors so homoiconic CL — where every defXXX form is the same list_lit node — is classified by reading definer symbols rather than tree-sitter node types. Co-authored-by: jansaasman <ja@franz.com>
Emit docstring-derived rationale nodes with file_type "rationale" instead of "code" so the cross-file resolution filter excludes them, keeping docstrings out of import and symbol-resolution edges. Co-authored-by: jansaasman <ja@franz.com>
Classify .lisp/.cl/.lsp/.asd as code and route them to the Common Lisp extractor so ASDF systems and source files build into the graph. Co-authored-by: jansaasman <ja@franz.com>
Declare the Common Lisp grammar under optional-dependencies, matching the policy for other niche grammars, so the core install stays lean and Lisp users opt in with the commonlisp extra. Co-authored-by: jansaasman <ja@franz.com>
Add the sample.lisp fixture and the Common Lisp extraction tests covering packages, classes, defuns, generics, macros, method specializers, inheritance, imports, custom definers, reader conditionals, and docstring rationale typing. Co-authored-by: jansaasman <ja@franz.com>
The latest tree-sitter-commonlisp release returns an int pointer that the tree-sitter runtime only accepts via a deprecated path, so extracting Lisp spewed a DeprecationWarning per file. Suppress that one warning at the construction site so Lisp extraction runs clean. Co-authored-by: jansaasman <ja@franz.com>
Move the Common Lisp extraction tests into the language-extractor suite where the project keeps per-language coverage, so they sit with their peers rather than in the general multi-language file. Co-authored-by: jansaasman <ja@franz.com>
Surface Common Lisp in the supported-extensions table and add the commonlisp optional-extra row so users know the grammar exists and how to install it. Co-authored-by: jansaasman <ja@franz.com>
extract.py is being decomposed one language per module (Graphify-Labs#1212), so a newly added language belongs in the package rather than adding another 500 lines to the file being emptied. The facade re-export and the LANGUAGE_EXTRACTORS entry keep every existing importer and the registry identity guard working unchanged. The extractor body is unchanged; only the import plumbing moved.
The Common Lisp branch carried a second copy of the "Main extract and collect_files" banner and _check_tree_sitter_version. Python let it pass because the two definitions are identical, so nothing failed, but the file should define it once.
The grammar is an optional extra, so a default install does not have it, and these tests called the extractor unguarded: absent grammar meant a hard failure rather than a skip. Every other optional-grammar language already guards this way. Matters for a fresh checkout and for any lane that installs from the lock file, where the extra is not present at all.
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
This pull request adds a new Common Lisp extractor. It introduces graphify/extractors/commonlisp.py (backed by the tree-sitter-commonlisp grammar) and wires it into the extractor registry, the _DISPATCH table in extract.py, and the CODE_EXTENSIONS set in detect.py for .lisp, .cl, .lsp, and .asd files. Supporting changes update the README to document the new commonlisp optional install and file-type coverage (bumping the grammar count from 36 to 37), and add a corresponding test suite for the Common Lisp language extraction. The surface area spans detection, extraction dispatch, the new extractor module, docs, and tests.
No blocking issues surfaced.
Analysis details — impact, health, verification
Impact & health
graphify review
Impact — 1844 function(s) in the blast radius of 785 changed node(s).
Health — regressions introduced by this change:
- new offender:
extract_commonlisp()(Ca=25 Ce=1) — 25 callers depend on it (afferent coupling) - new offender:
_process_form()(Ca=1 Ce=10) — fans out to 10 callees (efferent coupling) - new offender:
_handle_def_form()(Ca=1 Ce=6) — fans out to 6 callees (efferent coupling) - new offender:
_handle_defpackage()(Ca=1 Ce=6) — fans out to 6 callees (efferent coupling)
Verification — 1844 function(s) need re-proving (callee-first):
graphify_extract_xaml_inferred_viewmodel_names_add → graphify_extract_safe_extract → graphify_extract_safe_extract_with_xaml_root → graphify_extract_node_label_key → graphify_extract_lang_is_case_insensitive → graphify_extract_lang_family → graphify_extract_is_top_level_function_definition → graphify_extract_rewire_unique_stub_nodes → graphify_extract_file_node_id → graphify_extract_repoint_python_package_imports …
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not enforced this run (re-run with --prove to verify the blast radius)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1686 function(s) in blast radius — re-run with
--proveto formally verify them
· 4 grounded finding(s) anchored inline below.
| }) | ||
|
|
||
|
|
||
| def extract_commonlisp(path: Path) -> dict: |
There was a problem hiding this comment.
extract_commonlisp()
25 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| t = _text(node) | ||
| return t.lstrip(":").lstrip("#:") | ||
|
|
||
| def _handle_defpackage(node) -> None: |
There was a problem hiding this comment.
_handle_defpackage()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| idx += 1 | ||
| return children[idx:] | ||
|
|
||
| def _handle_def_form(node, def_keyword: str) -> None: |
There was a problem hiding this comment.
_handle_def_form()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return False | ||
| return sym.startswith("def") or sym.startswith("define-") | ||
|
|
||
| def _process_form(top) -> bool: |
There was a problem hiding this comment.
_process_form()
fans out to 10 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
1 participant
The ask
Do you want Common Lisp as a supported language in graphify? A yes or a no is all I need.
If the answer is no, I will close this and stop. Either answer is genuinely useful to us, and
thank you for the attention it takes to give one.
Where it comes from
This is not a fresh implementation. It descends directly from #172 by @jansaasman, opened
on 9 April, and the history still carries his commits as its base:
Add Common Lisp support (language #21), 9 AprilCommon Lisp: walk wrapper macros, reader conditionals, more definer forms, 10 AprilFix edge direction lost in JSON, Cypher, HTML, and Canvas exports, 10 AprilThat work established the extraction approach: which forms count as definitions, how to walk
wrapper macros so definitions nested inside them are not lost, and how reader conditionals are
handled. Those decisions are his and they are still the ones in force here. What has been added
on top is consolidation rather than redesign:
emits
If Common Lisp lands, the credit for the design belongs upstream of this PR.
How it has tracked the codebase since April
graphify has moved a long way in the four months since #172 was opened, and this branch has been
rebased forward twice rather than left to rot:
spanning the 0.8.35 to 0.9.29 releases
The rebases were not mechanical. Each one surfaced real conflicts where the two sides had grown
into the same place: the code extension set had gained
.mts,.cts, CUDA, Metal and othersalongside the four Lisp ones; the optional extras table had gained
pascal; the README languagerow had grown from 29 grammars to 36. In every case the upstream side was taken and the Lisp
addition re-applied on top, so nothing of yours was reverted by the rebase.
Reorganised to match where extract.py is going
The most significant change since #172 is structural, and it is the reason this PR looks
different from that one.
extractors/MIGRATION.mdstates thatgraphify/extract.pyis being split into thegraphify/extractors/package, one language per PR, tracking #1212. Twenty seven languagemodules already live there. #172 predates that split and added its extractor to
extract.py,which was correct at the time and is now against the grain.
So the extractor has been moved to
graphify/extractors/commonlisp.py, following the playbook inMIGRATION.md rather than inventing a shape:
graphify.extractors.base, never fromgraphify.extract, so the dependency direction the package requires is preservedgraphify.extract, kept alphabeticalLANGUAGE_EXTRACTORSinextractors/__init__.pytest_extractors_registry.py. The registry sweep there now coverscommonlispautomatically.The effect on the file being emptied is the point:
extract.pygrows by five lines rather thanby five hundred. The extractor is 492 lines and all of it lands in the new package.
If a new language should go somewhere other than
extractors/, say so and I will move it. Themigration document is written for relocating existing extractors and does not state where new
ones belong, so this is an inference from direction of travel, not a claim about your intent.
What it is
.lisp,.cl,.lspand.asd, viatree-sitter-commonlisp.macros and variables;
contains,calls,inherits,specializesandimportsedges.commonlispoptional extra, so a defaultuv tool install graphifyyisunchanged, matching how
dm,terraformandpascalalready behave.tests/test_languages.pyagainsttests/fixtures/sample.lisp, each skipped whenthe grammar is absent, in the same style as the existing
dmguard.not also fail on v8 unpatched.
⚠ One thing to flag rather than let you discover:
uv.lockhas notree-sitter-commonlispentry, while the other optional grammars do. CI installs with
--frozen, so the grammar will notbe present there and the Lisp tests will skip rather than run. They will not fail, but they will
also not be proving anything on your side until the lock is regenerated. I have not touched the
lock file, since hand editing it seemed worse than naming the gap.
Why a ruling rather than a review
Two Common Lisp PRs are open, #172 since April and this one since June, and neither has drawn a
comment. I do not read that as neglect: your recently merged PRs are outside contributions landing
within a day or two. It reads instead like these two are asking a different kind of question. Not
"is this code correct" but "does graphify want Common Lisp, and another optional grammar
dependency, at all". That is a scope call, it is yours, and one word settles it.
We maintain a fork in the meantime, so a no costs us nothing and is a perfectly good answer. What
does cost something is a third rebase against silence.