Test automation v1: theseus-qemu test, kernel test runner, TDD workflow#32
Open
emilf wants to merge 5 commits into
Open
Test automation v1: theseus-qemu test, kernel test runner, TDD workflow#32emilf wants to merge 5 commits into
emilf wants to merge 5 commits into
Conversation
added 5 commits
June 11, 2026 02:02
- Add feature to kernel/Cargo.toml (feature-gated, no-op in release) - Add : Test struct, run_kernel_tests() function - Wire test runner into after_high_half_entry() in environment.rs, before idle loop - Add kernel/src/testing module declaration to lib.rs (cfg-gated) - Propagate feature through bootloader/Cargo.toml as theseus-kernel/kernel-tests - Add QEMU_PANIC (2) exit code to shared/src/constants.rs Tests return Result<(), &'static str>. All OK → qemu_exit_ok!(). Any FAIL → qemu_exit_error!() Output goes via kernel logging (debugcon port 0xE9).
- Add Cmd::Test(TestArgs) subcommand to theseus-qemu CLI - TestVerdict enum: Pass(0), Fail(1), Panic(2), Timeout(3) - run_kernel_tests(): builds with FEATURES=kernel-tests via make all, runs QEMU headless (Min profile), parses TEST_RESULT marker from output - Saves .test-output.log on non-PASS exit; --print flag for verbose output - --timeout <secs> (default 60), --no-build flag - make test convenience target in Makefile Verdict is determined primarily by TEST_RESULT: PASS/FAIL marker in QEMU debugcon output, since QEMU's isa-debug-exit device always does exit((val << 1) | 1) and cannot produce exit code 0.
- New docs/plans/agent-test-automation.md: comprehensive design doc covering exit code scheme, TDD workflow, kernel test infrastructure, tool subcommand - Update docs/plans/testing.md: reflect implemented state, include current code examples, agent guidance, and TDD workflow - Update docs/map.md: add entries for testing.md and agent-test-automation.md - Add TDD workflow instruction block to docs/agent-prompt-template.md - Add test automation section to AGENTS.md (exit codes, TDD, cardinal rule)
…on string parsing The kernel test runner now writes dedicated raw values to port 0xf4: - QEMU_EXIT_TEST_PASS (3) — transforms via (val<<1)|1 to QEMU exit 7 - QEMU_EXIT_TEST_FAIL (4) — transforms to QEMU exit 9 The panic handler still uses QEMU_ERROR (1) which transforms to exit 3. theseus-qemu test maps QEMU exits 7/9/3/124 to verdicts PASS/FAIL/PANIC/TIMEOUT (exit codes 0/1/2/3). No debugcon string parsing needed. Also added qemu_exit_test_pass!() and qemu_exit_test_fail!() macros. Also updated exit code section in the plan doc and kernel source comments.
…NIC/TIMEOUT scenarios This implements comprehensive testing of the test automation infrastructure: - Adds real kernel tests that verify ISA debug exit constants, transform formulas, and port assignments - Introduces build-time scenario selection via Cargo features: - kernel-test-scenario-fail → one test fails, expects exit code 1 (FAIL) - kernel-test-scenario-panic → one test panics, expects exit code 2 (PANIC) - kernel-test-scenario-hang → kernel hangs, expects exit code 3 (TIMEOUT) - default (no scenario flag) → all tests pass, expects exit code 0 (PASS) - Adds --features arg to theseus-qemu test subcommand to pass extra cargo features through to the kernel build - Adds Makefile convenience targets: test-fail, test-panic, test-hang, test-all-verdicts (runs all 4 scenarios with exit code verification) Verified each verdict produces the correct exit code end-to-end through QEMU.
Owner
Author
|
I just tried running the tests and I have an observation. If I run cargo run -p theseus-qemu -- test --features kernel-test-scenario-hang it doesn't time out... the tests have to have a timeout by default, otherwise it can cause the process to hang and that makes whoever is running the tests to wait forever. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements a complete test automation infrastructure for TheseusOS, enabling agents to get a structured PASS/FAIL/PANIC/TIMEOUT verdict from a single command without parsing log output.
Changes
Kernel side (test-automation-v1)
kernel-testsfeature (feature-gated, no-op in release builds)Teststruct andrun_kernel_tests()function — tests returnResult<(), &'static str>after_high_half_entry()before the idle looptestingmodule declaration (cfg-gated)QEMU_PANIC (2)exit codeTooling side
testsubcommand withTestVerdictenum, output parsing, timeout supportmake testconvenience targetDocumentation
Exit Code Scheme
TDD Workflow (Mandatory)
cargo run -p theseus-qemu -- test→ exit 0 (PASS)cargo run -p theseus-qemu -- test→ exit 1 (FAIL)cargo run -p theseus-qemu -- test→ exit 0 (PASS)Verdict: PASS (exit code 0)Verification
make all(without features) builds cleanly ✓make all FEATURES=kernel-testsbuilds with tests ✓cargo run -p theseus-qemu -- testreturns TEST PASS (exit 0) with zero tests ✓cargo build -p theseus-kernelwithout features is unchangedNotes
isa-debug-exitdevice transforms exit codes viaexit((val << 1) | 1), soqemu_exit_ok!()produces QEMU exit 1. The test runner handles this by parsingTEST_RESULT: PASS/TEST_RESULT: FAILfrom the debugcon output instead of relying on raw exit codes.