runner: failing set_up_before_script is inconsistent and can silently lose tests

[Chemaclass]

Context

Found while debugging build.sh --verify on a jq-less ubuntu box (#834): tests/acceptance/bashunit_report_json_test.sh reported "Tests: 5" declared, 4 total ran, 3 failed with 0 assertions and no failure details, no hook-failure message. The root event was simply set_up_before_script returning non-zero (its last statement is command -v jq >/dev/null 2>&1 && JQ_AVAILABLE=true on a box without jq).

A failing set_up_before_script currently produces three different outcomes depending on incidental properties of the test file. All measured on 0.41.0+ (main at 559180e), sequential mode:

Repro matrix

A. Plain file (no set -euo pipefail), hook ends with failing command:

printf '#!/usr/bin/env bash
function set_up_before_script() {
  command -v definitely_missing_tool >/dev/null 2>&1
}
function test_a() { assert_true true; }
function test_b() { assert_true true; }
' > /tmp/a_hook.sh
./bashunit --no-parallel /tmp/a_hook.sh

Result: Hook 'set_up_before_script' failed with exit code 1, both tests marked failed, suite continues. This is the good path — visible, attributed, non-fatal.

B. File with set -euo pipefail, hook ends with cmd && var=true guard:

Same fixture plus set -euo pipefail and local X=false; command -v missing && X=true in the hook. Result: hook failure silently ignored — all tests pass, no message. Inconsistent with A.

C. The real-world case (report_json on a jq-less box, docker ubuntu:24.04):

docker run --rm -v "$PWD":/repo -w /repo ubuntu:24.04 bash -c \
  './bashunit --no-parallel tests/acceptance/bashunit_report_json_test.sh tests/acceptance/bashunit_bench_test.sh'

Result: header declares Tests: 5, summary says 4 total; the 3 jq-gated tests are counted failed with 0 assertions, but there is no hook-failure message and no per-test failure detail block — the suite output contains zero explanation. Mechanism: the hook's variable assignments (JQ_AVAILABLE=false) are lost when the hook fails, so each test hits [ "$JQ_AVAILABLE" = false ] with the var unset, dies under set -u inside its $() subshell before any assertion, and surfaces as a detail-less failure. One declared test also vanishes from the totals.

During the full-suite run in that environment the suite output additionally truncated after this file (no later Running … lines, no summary in the captured output, exit 1) — reproduce with ./bashunit tests/ in the same container; the two-file invocation above does not truncate, so the abort needs the full-suite context to trigger.

Why this matters

A test framework's core promise is that untested code is reported, not skipped in silence. Outcome C fails that promise twice (no message, wrong declared count); outcome B fails it completely (hook failure invisible). The #834 hunt burned a full debug cycle on exactly this silence.

Acceptance criteria

• One defined behavior for a failing set_up_before_script, regardless of set -euo pipefail in the test file or the shape of the hook's last statement: every test in the file is marked failed (or errored) with the existing Hook 'set_up_before_script' failed message, and the suite continues to the next file
• Declared test count matches executed+reported count when a hook fails (no vanishing tests)
• Failure details (deferred failure block) are printed for hook-failure casualties — never a bare N failed with no explanation
• A test that reads a hook-assigned variable which was never set (because the hook died) must still surface as an attributed failure, not a detail-less set -u subshell death
• Full-suite sequential run in a jq-less environment completes with a summary (regression check for the truncation)
• Same behavior in --parallel
• tear_down_after_script audited for the same inconsistency while in there

Constraints

• Bash 3.0+ in src/ (no [[ ]], declare -A, ${var,,}, BASHPID)
• Fork budget: no new per-test or per-file forks (tests/acceptance/bashunit_*forks_test.sh gate on 3 platforms)
• TDD: repro fixtures A/B/C as failing tests first; fixture files must not match *[tT]est.sh (discovery snapshots)
• Relevant code: bashunit::runner::run_set_up_before_script and its caller in src/runner.sh; the existing "Hook … failed" path shows where outcome A is produced

Verification

./bashunit tests/ && ./bashunit --parallel --simple --strict tests/
make sa && make lint
docker run --rm -v "$PWD":/repo -w /repo ubuntu:24.04 bash -c './bashunit --no-parallel tests/'  # jq-less: completes with summary

[Chemaclass]

Done in #838 (merged). One defined behavior for failing lifecycle hooks across bash 3.2/5.x; strict files can no longer abort the suite; 13 latently strict-broken tests fixed.