Skip to content

[codex] add evidence-first TrustDocument extraction#17

Open
JamesbbBriz wants to merge 2 commits into
mainfrom
work/oss-evidence-sdk-polish
Open

[codex] add evidence-first TrustDocument extraction#17
JamesbbBriz wants to merge 2 commits into
mainfrom
work/oss-evidence-sdk-polish

Conversation

@JamesbbBriz

@JamesbbBriz JamesbbBriz commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Problem

DocTruth's OSS value was drifting too parser-only: parsing improvements existed, but the SDK/CLI did not expose a complete evidence-first extraction workflow that users can inspect, replay, and audit.

What Changed

  • Adds a public TrustDocument contract and JSON writer/parser surface for parser output.
  • Makes OpenDataLoader the default PDF-backed TrustDocument parse path while preserving legacy parsed-section JSON compatibility.
  • Adds doctruth profile for local parser latency/heap profiling.
  • Updates doctruth extract to write a complete run package:
    • trust-document.json
    • result.json
    • audit.json
    • manifest.json
  • Links audit derivations back to TrustDocument evidence units with doctruth:sourceDocId and doctruth:sourceUnitId.
  • Adds SDK/CLI evidence-first JSON extraction:
    • JsonExtractionBuilder.withEvidenceFirst()
    • DocumentJsonExtractionBuilder.withEvidenceFirst()
    • doctruth extract --evidence-first
  • Wraps schema leaves as { value, exactQuote }, validates the wrapped provider response, unwraps result JSON, and cites the supplied exact quote.
  • Recursively requires citations for schema leaf paths by default, including array paths such as items[].name matching items[0].name.
  • Resolves local JSON Schema $ref / $defs entries for evidence-first schema wrapping and CLI required-field collection.
  • Keeps evidence-first unwrapping schema-aware so business objects with fields named value and exactQuote are not mistaken for evidence wrappers.
  • Removes OpenDataLoader parser root JUL logger mutation during parse.
  • Avoids absolute source-path leakage in extract manifest.json; records sourceFilename and sourceSha256 instead.
  • Updates public API snapshot, docs, NOTICE, and coverage tests.

Security / Privacy Impact

  • No telemetry, background network listener, hosted callback, or remote state is added.
  • Extraction run artifacts are local files written only to the selected output directory.
  • Extract manifests no longer include absolute local source paths.
  • No secrets or fixture corpora are added.
  • New OpenDataLoader dependency license obligations are reflected in NOTICE.

Fixture / Migration Notes

  • Public API snapshot intentionally changes for TrustDocument*, CitationSource, and withEvidenceFirst().
  • Existing Citation constructors remain available; TrustDocument source identity is exposed via optional Citation.source().
  • doctruth parse --json remains legacy parsed-section JSON; --format json writes TrustDocument JSON.

Tests Run

mvn -B -ntp spotless:apply spotless:check checkstyle:check -DskipTests
mvn test
mvn verify -P recorded
git diff --check

Results:

  • spotless:check checkstyle:check: passed, 0 Checkstyle violations.
  • mvn test: 766 tests before resolver coverage test, then 769 tests after resolver coverage test; 0 failures, 0 errors, 0 skipped.
  • mvn verify -P recorded: passed; unit tests 769/0/0/0, integration tests 8 run with 3 skipped, JaCoCo checks met.
  • git diff --check: passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant