Add importer support for newer OWASP standards for issue #471#4
Open
Bornunique911 wants to merge 1 commit into
Open
Add importer support for newer OWASP standards for issue #471#4Bornunique911 wants to merge 1 commit into
Bornunique911 wants to merge 1 commit into
Conversation
Bornunique911
pushed a commit
that referenced
this pull request
Jun 24, 2026
OWASP#928) * feat(module-b): add Pydantic v2 schemas + hashing for Module A input contract Establishes the data contract Module B consumes from Module A. ChangeRecord is a Pydantic v2 model matching A's actual emission shape: nested source (discriminated union on type for github/rss), span (chunk position + heading_path + char/line offsets), and locator (addressing scheme). Internal models ClassifyResult and QueuePayload prep for later stages. hashing.py provides normalize_text + compute_content_hash since Module A does not emit content_hash; B computes its own (SHA-256 of normalized text) for use as the knowledge_queue dedup key. 22 unittest cases cover the round-trip, the discriminated union, hash determinism, normalization rules, code-fence preservation, and idempotency. Full make test: 271 passing, no regressions. Part of GSoC 2026 OpenCRE Scraper & Indexer (Project OIE) Module B. * feat(module-b): add Module A mock fixture + generated JSON Schema artifact module_a_mock.jsonl: Module A's canonical 20-record mock shared 2026-05-29, saved as JSONL (one record per line per the contract). Becomes a permanent integration-test fixture for B's parser and a reference shape for the Module A contributor. module_a_contract.schema.json: JSON Schema generated from B's Pydantic ChangeRecord model via model_json_schema(). 246 lines covering all four nested types (ChangeRecord, GithubSource, RssSource, Span, Locator). Source of truth for cross-module CI validation. Part of GSoC 2026 OpenCRE Scraper & Indexer (Project OIE) Module B. * feat(module-b): add OWASP harvester, labeling TUI, and labeled dataset build_labeled_dataset.py: PyGithub-based harvester that acts as Module A's stand-in for producing benchmark data. Fetches recent commits from 4 OWASP repos (WSTG, ASVS, CheatSheetSeries, SAMM), applies the contract's normalization rules, splits into chunks at markdown heading boundaries with a fence-aware stack-based walker that tracks heading_path + char/line offsets, and emits records in Module A's actual nested shape. Pluggable via GITHUB_TOKEN env var. Reproducible: python scripts/build_labeled_dataset.py regenerates the candidate set. label_dataset.py: resumable interactive TUI for manual classification. Atomic-writes labeled_data.json after every keystroke; lookup by chunk_id for resume. Embeds the recall-first definition (agreed with maintainer 2026-06-01) so labelers see the rule front-of-mind: KNOWLEDGE for any chunk with security signal, NOISE only for pure organizational content. candidate_commits.json: 100 records, 25 per repo, all Pydantic-valid against ChangeRecord. 90/100 have non-empty heading_path; 10 multi-chunk artifacts captured. labeled_data.json: 100/100 labeled by hand under the recall-first rule. Distribution 55 KNOWLEDGE / 40 NOISE / 5 UNCERTAIN. Per-repo skew is visible: CheatSheetSeries 92% K, SAMM 0% K (the SAMM commits sampled landed entirely on Website/Sponsorship/meetings paths -- empirical input for Week 2's noise_patterns.yaml). Part of GSoC 2026 OpenCRE Scraper & Indexer (Project OIE) Module B. * style(module-b): apply Black formatting to Week 1 files Super-Linter (Black 24.4.2) flagged 4 files in the previous push. Applied `black` (same pinned version) to bring them in line with the repo's formatting standard. Cosmetic changes only: blank lines around section-separator comments, one multi-line dict join. No behavior or test changes -- `make test` remains 271 passing, 1 skip. * chore(module-b): address CodeRabbit Week 1 review comments - Sort __all__ lists in hashing.py and schemas.py to satisfy Ruff RUF022. - Declare JSON Schema dialect ($schema = draft 2020-12, which is what Pydantic v2 model_json_schema() emits) on the contract artifact. - Wrap load_labeled() in scripts/label_dataset.py with try/except so a corrupted labeled_data.json prints an actionable hint instead of a raw JSONDecodeError stack trace. Deferred to Week 2 (will be addressed when we touch the harvester): - chunker should also track <pre> open/close, not just ``` fences - _split_chunk_by_size cursor arithmetic assumes \\n\\n separator even on hard-split sub-chunks Tests: 271 passing, 1 skip (unchanged). Black: clean. * feat(module-b): add Stage 1.5 sanitize.py vendored from TRACT Defensive text cleanup (PDF ligatures, zero-width chars, HTML, hyphenation). Vendored from rocklambros/TRACT under CC0; drops their whitespace-collapse step so structure (newlines, paragraphs) is preserved for Module B's LLM. 26 unit tests, all passing. * feat(module-b): add Stage 1 regex_filter + noise_patterns.yaml Path-based filter with extension/filename/glob deny rules and allow_overrides. Patterns are deliberately conservative under the recall-first labeling rule. 15 unit tests including >=90% rejection / 0% false-positive acceptance criteria. * fix(module-b): chunker tracks <pre> blocks; correct hard-split cursor math Addresses CodeRabbit comments #4 and #5 on the Week 1 PR. * chore(module-b): address CodeRabbit Week 2 review comments * chore(module-b): address Week 2 maintainer review on noise_patterns.yaml --------- Signed-off-by: Manshu Saini <149303743+manshusainishab@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue
This branch addresses:
#471Mapping: to more owasp resourcesProblem
OpenCRE does not yet include parser/import support for several newer OWASP resources that are useful for mapping CREs to current guidance.
Without importer support for these resources:
Solution
This branch adds parser and data support for newer OWASP resources, including:
The implementation uses explicit parser and data files so the imported resources remain maintainable and reviewable.
This branch also adds the required parser wiring so these resources can be imported through the existing OpenCRE flow.
Testing
Executed focused parser tests: