Add systematic type-matrix and fuzz tests for expression functions - #350
Open
davidkhk wants to merge 12 commits into
Open
Add systematic type-matrix and fuzz tests for expression functions#350davidkhk wants to merge 12 commits into
davidkhk wants to merge 12 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Systematic test-quality improvements for the Expression library: type-matrix tests across all five function categories, a property-based fuzz suite, and supporting infrastructure. No
lib/changes — test code, docs, and one CI workflow only.Tests document current V1 engine behavior, they do not endorse it: where a function crashes on unexpected input, the test pins the exact exception (
assert_raise+ a# Known crash behavior, documented not endorsed:comment) so any future change is deliberate, not accidental.What's included
stream_datadep;Expression.Test.TypeTestMatrix(canonical per-type sample values + evaluation helpers);Expression.Test.FuzzHelpers(StreamData generators +assert_no_crash); support modules compiled viaelixirc_pathsfor:test.__value__maps / unicode / edge cases. Pilot tests forupper,lower,abs,round,date.Expression.Test.CrashSafe, consumed by both a deterministic classification test (runs in the normal suite, pins the set against the full type matrix on every CI build) and the fuzz suite (random exploration on top).expression_fuzz_test.exs, tagged:fuzz, excluded from the default suite).TESTING.mddocumenting the approach, helpers, the context-coercion gotcha, and how to add tests for new functions.CI: fuzz runs on a schedule, not as a merge gate
Property tests use a fresh random seed each run and exist to discover new crashing inputs, so they can legitimately go red — a poor fit for branch protection.
.github/workflows/fuzz.ymlruns them nightly (plus a manual "Run workflow" button) withFUZZ_MAX_RUNS=1000, so they never block PRs ordevelopmerges. On failure, ExUnit prints the seed and StreamData prints the shrunk input; reproduce withmix test --only fuzz --seed <N>.Real defects surfaced (documentation-only here, pinned as current behavior)
trimandfirstare not implemented — they returnERROR: "... is not implemented."strings despite being FLOIP functions.char/1is byte-based, not codepoint-based —char(233)returns invalid UTF-8<<233>>instead of"é", and disagrees withunichar.find/has_membercrash on non-lists (Protocol.UndefinedError) while sibling enum functions degrade to error maps.to_stringasymmetries between sibling functions (url_encode/url_decode,remove_first_word/remove_last_word).Candidates for follow-up hardening; adding a function to
Expression.Test.CrashSafeis the signal it's been fixed.Verification
FUZZ_MAX_RUNS=1000(20,000 evaluations)mix format --check-formattedclean ·mix credo --strictexit 0 ·MIX_ENV=test mix compile --warnings-as-errorsclean