PyReprism

PyReprism is a Python framework that helps researchers and developers the task of source code preprocessing. With PyReprism, you can easily match, extract, count, and remove comments, whitespaces, operators, numbers and other language specific constructs from over 150 programming languages and file extensions.

Install

pip install PyReprism

Quick Usage

The simplest entry point is the top-level API. Pass lang as a language name ("python"), a file extension (".py"), or a language class.

import PyReprism as pr

source = """
# single line comment
x = 5 + 6
print(x)  # inline comment
"""

pr.remove_comments(source, lang="python")   # -> code with comments stripped
pr.extract_comments(source, lang="python")  # -> ['# single line comment', '# inline comment']
pr.count_comments(source, lang="python")    # -> 2

Every construct supports the same match / extract / count / remove verbs:

code = 'total = price * 42 + tax  # compute'

pr.extract_numbers(code, lang="python")      # -> ['42']
pr.extract_keywords(code, lang="python")     # -> []
pr.extract_identifiers(code, lang="python")  # -> ['total', 'price', 'tax', 'compute']
pr.remove_numbers(code, lang="python")       # -> 'total = price *  + tax  # compute'
pr.extract_strings('s = "hi"', lang="python")  # -> ['"hi"']

match_* returns positioned Tokens (type, value, start, end, line):

for tok in pr.match_comments(source, lang="python"):
    print(tok.line, tok.value)

Tokenize

tokenize() returns a lossless, typed token stream (comment/string/number/ keyword/identifier/operator/whitespace/other):

for tok in pr.tokenize("x = 5 + y  // c", lang="clike"):
    print(tok.type, repr(tok.value))

Pipelines

Chain removal steps in one call:

pr.preprocess(source, lang="java",
              steps=["comments", "strings", "whitespace"])

Valid steps: comments, strings, numbers, operators, keywords, whitespace.

Code metrics

s = pr.stats(source, lang="python")
s.code_lines, s.comment_lines, s.blank_lines
s.comment_to_code_ratio, s.comment_density
s.as_dict()   # includes per-token-type counts, ready for JSON / dataframes

Complexity metrics (computed from the token stream):

pr.halstead(source, lang="python").volume        # Halstead volume/difficulty/effort/bugs
pr.cyclomatic_complexity(source, lang="python")  # approximate McCabe complexity
pr.maintainability_index(source, lang="python")  # 0–100 (higher is better)
pr.code_metrics(source, lang="python")           # everything above in one dict

On the CLI: pyreprism stats --full file.py (add --json for machine output).

Similarity & clone / plagiarism detection

Winnowing k-gram fingerprints over the normalized token stream, so matches survive variable renaming and literal changes (Type-2 clones):

from PyReprism import fingerprints as fp

fp.similarity(code_a, code_b, "python")     # 0.0–1.0 (Jaccard of fingerprints)
fp.containment(code_a, code_b, "python")    # how much of A appears in B

index = fp.FingerprintIndex()               # many-to-many clone detection
index.add_paths("submissions/")
index.similar_pairs(threshold=0.7)          # -> [(file_a, file_b, score), ...]

On the CLI: pyreprism similarity a.py b.py and pyreprism clones submissions/ --threshold 0.7.

N-grams & code "naturalness"

Token n-grams (over token text or, structurally, over token types) and an n-gram language model that measures how predictable/"natural" code is (Hindle et al.):

from PyReprism import ngrams

ngrams.ngram_counts(source, "python", n=3).most_common(10)
ngrams.ngrams(source, "python", n=2, types=True)      # structural n-grams

model = ngrams.train("corpus/", n=3)                  # train on a code corpus
model.perplexity(ngrams.token_sequence(source, "python"))   # lower = more natural
model.save("model.json")

On the CLI: pyreprism ngrams file.py -n 3 --top 20 and pyreprism perplexity --train corpus/ file.py.

Normalization for ML / clone detection

Canonicalize code so that only its structure remains — rename identifiers to VAR1, VAR2, …, mask string/number literals, and drop comments:

pr.normalize("total = price * 42  # cost", lang="python")
# -> 'VAR1 = VAR2 * 0'

Every part is toggleable (rename_identifiers, mask_numbers, mask_strings, drop_comments, collapse_whitespace). To strip comments while keeping line numbers stable (useful for tools that map back to source):

pr.blank_comments(source, lang="python")   # comment content removed, newlines kept

Accuracy: choose a backend

By default PyReprism uses its own zero-dependency regex engine (fast, no installs). For higher accuracy — e.g. correctly ignoring a # inside a string — pass engine="pygments" (install the optional extra with pip install pyreprism[accurate]):

src = 'url = "http://x#frag"  # real comment'

pr.remove_comments(src, lang="python")                     # regex (default)
pr.remove_comments(src, lang="python", engine="pygments")  # keeps the URL, drops the comment

engine accepts "regex" (default), "pygments", or "auto" (use pygments if installed, else regex). It works on every operation — remove_*, extract_*, count_*, tokenize, normalize, stats, preprocess — and on the CLI via --engine.

Batch / whole-directory processing

Analyze or transform an entire source tree (junk folders like .git, node_modules, venv are skipped automatically):

from PyReprism import batch

report = batch.analyze("myproject/")     # walk the tree, compute metrics
report.totals()                          # aggregate line/token counts
report.by_language()                     # per-language breakdown
report.to_json(); report.to_csv()        # export for dataframes / notebooks

# Bulk-transform into a mirrored output directory:
batch.transform("myproject/", lambda text, lang: lang.remove_comments(text),
                output="stripped/")

Process diffs / pull requests

Parse a unified (git) diff and analyze the changed code per file, language-aware:

from PyReprism import diffs

d = diffs.parse(diff_text)          # git diff / .diff / .patch text

report = diffs.diff_stats(d)        # churn split into code vs comment vs blank
report.totals(); report.to_csv()

diffs.cosmetic_files(d)             # files whose change is comment/whitespace-only

f = d.files[0]
f.language                          # detected from the file path
f.added_text(); f.removed_text()    # reconstructed changed code
f.normalize("added")               # canonicalize the added code for ML
f.extract_comments("added")

For accuracy where a comment/string spans a hunk boundary, set f.new_source / f.old_source to the full file contents (e.g. from git show) and the changed lines are classified against the whole file. On the CLI:

git diff | pyreprism diff --per-file      # churn report
git diff | pyreprism diff --json
git diff | pyreprism diff --cosmetic      # list comment/whitespace-only changes

Detect the language from a filename

cls = pr.detect_language(filename="src/main.go")   # -> Go language class

Language-independent whitespace normalization

pr.remove_whitespaces("x = 5 + 6\n\n\nprint(x)")  # -> 'x=5+6\nprint(x)'

You can also import a language class directly if you prefer:

from PyReprism.languages import Python
Python.remove_comments(source)

Command line

Installing PyReprism also provides a pyreprism command (works on stdin/stdout, files, and globs; the language is auto-detected from a file's extension):

pyreprism remove comments file.py
cat file.go | pyreprism remove comments --lang go
pyreprism extract comments "src/**/*.py" --json
pyreprism count comments file.py --lang python
pyreprism preprocess --steps comments,strings,whitespace file.java
pyreprism tokenize --json file.py
pyreprism stats --json file.py                 # line/token metrics
pyreprism normalize file.py                    # canonicalize for ML
pyreprism remove comments file.py --in-place   # rewrite in place
pyreprism scan myproject/ --csv                # aggregate metrics over a tree
pyreprism remove comments src/ --output out/   # bulk-transform a directory
pyreprism languages                            # list supported languages

Equivalently: python -m PyReprism ....

It can also be wired into pre-commit via the bundled .pre-commit-hooks.yaml (pyreprism-strip-comments, pyreprism-normalize-whitespace). These hooks rewrite files, so scope them with files:/exclude:.

Read the docs for more usage examples.

NB: The beta versions of PyReprism is still unstable, but we are working 24/7 to ensure the tool is usable.

How to Contribute

We invite you to help us build this tool and make it more extensive. Contribution is open to OSS community.

$ git clone https://github.com/unlv-evol/PyReprism.git
$ cd PyReprism

(Optional) It is suggested to make use of virtualenv. Therefore, before installing the requirements run:

$ python3 -m venv venv
$ source venv/bin/activate

Then, install the package in editable mode with its development tooling:

$ pip install -e ".[dev]"

For more information on how to contribute, read our contributing guidelines.

Issues

If you experience any issue, feel free to report it.