Skip to content

Add systematic type-matrix and fuzz tests for expression functions - #350

Open
davidkhk wants to merge 12 commits into
developfrom
test-quality-improvements
Open

Add systematic type-matrix and fuzz tests for expression functions#350
davidkhk wants to merge 12 commits into
developfrom
test-quality-improvements

Conversation

@davidkhk

Copy link
Copy Markdown
Contributor

Summary

Systematic test-quality improvements for the Expression library: type-matrix tests across all five function categories, a property-based fuzz suite, and supporting infrastructure. No lib/ changes — test code, docs, and one CI workflow only.

Tests document current V1 engine behavior, they do not endorse it: where a function crashes on unexpected input, the test pins the exact exception (assert_raise + a # Known crash behavior, documented not endorsed: comment) so any future change is deliberate, not accidental.

What's included

  • Infrastructure: stream_data dep; Expression.Test.TypeTestMatrix (canonical per-type sample values + evaluation helpers); Expression.Test.FuzzHelpers (StreamData generators + assert_no_crash); support modules compiled via elixirc_paths for :test.
  • Type-matrix tests, one file per category (string, number, date, logical, enum) — each function probed against nil / numbers / strings / booleans / lists / maps / complex __value__ maps / unicode / edge cases. Pilot tests for upper, lower, abs, round, date.
  • Crash-safe classification: the set of functions confirmed crash-safe lives once in Expression.Test.CrashSafe, consumed by both a deterministic classification test (runs in the normal suite, pins the set against the full type matrix on every CI build) and the fuzz suite (random exploration on top).
  • Property-based fuzz tests (expression_fuzz_test.exs, tagged :fuzz, excluded from the default suite).
  • TESTING.md documenting the approach, helpers, the context-coercion gotcha, and how to add tests for new functions.

CI: fuzz runs on a schedule, not as a merge gate

Property tests use a fresh random seed each run and exist to discover new crashing inputs, so they can legitimately go red — a poor fit for branch protection. .github/workflows/fuzz.yml runs them nightly (plus a manual "Run workflow" button) with FUZZ_MAX_RUNS=1000, so they never block PRs or develop merges. On failure, ExUnit prints the seed and StreamData prints the shrunk input; reproduce with mix test --only fuzz --seed <N>.

The schedule trigger only activates once this file is on the default branch. After merge it can be exercised immediately via workflow_dispatch.

Real defects surfaced (documentation-only here, pinned as current behavior)

  • trim and first are not implemented — they return ERROR: "... is not implemented." strings despite being FLOIP functions.
  • char/1 is byte-based, not codepoint-basedchar(233) returns invalid UTF-8 <<233>> instead of "é", and disagrees with unichar.
  • find / has_member crash on non-lists (Protocol.UndefinedError) while sibling enum functions degrade to error maps.
  • to_string asymmetries between sibling functions (url_encode/url_decode, remove_first_word/remove_last_word).

Candidates for follow-up hardening; adding a function to Expression.Test.CrashSafe is the signal it's been fixed.

Verification

  • Default suite: 711 tests + 560 doctests, 0 failures
  • Fuzz suite: 20 properties, 0 failures — green at FUZZ_MAX_RUNS=1000 (20,000 evaluations)
  • mix format --check-formatted clean · mix credo --strict exit 0 · MIX_ENV=test mix compile --warnings-as-errors clean

@davidkhk davidkhk self-assigned this Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant