feat: swe bench scorer#342
Open
tianmu-li wants to merge 3 commits into
Open
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for SWE-bench accuracy evaluation by adding a new accuracy-only SWEBench dataset, a SWEBenchScorer that runs evaluations using mini-swe-agent in an isolated environment, and associated configuration templates, tests, and runbooks. Feedback on the changes focuses on improving the robustness of the SWEBenchScorer implementation, specifically by safely handling missing or null values when parsing configuration templates, benchmark configurations, and evaluation results, as well as gracefully handling cases where the Docker binary is missing from the system's PATH during preflight checks.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
2 times, most recently
from
aa7f559 to
bb9b307
Compare
There was a problem hiding this comment.
Pull request overview
Adds first-class SWE-bench accuracy support to the benchmarking system by introducing an external (subprocess-driven) scorer and a predefined SWE-bench dataset, plus wiring in config/schema handling and examples to run the workflow.
Changes:
- Introduces
SWEBenchScorer(external evaluation viauv run --project ... mini-extra+swebenchharness) and integrates it into the scoring/benchmark lifecycle (preflight, phase skipping, reporting). - Adds predefined
swe_benchdataset with caching + anACCURACY_ONLYguard to prevent using it as a performance dataset. - Updates schema/templates/examples and adds unit tests for the scorer + dataset + benchmark wiring.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Locks pyyaml as a direct dependency. |
| pyproject.toml | Adds pyyaml==6.0.3 to runtime dependencies. |
| src/inference_endpoint/evaluation/scoring.py | Adds scorer preflight hooks, shared subprocess logging helper, and new SWEBenchScorer implementation. |
| src/inference_endpoint/dataset_manager/predefined/swe_bench/init.py | Adds SWE-bench predefined dataset with HF download + parquet cache and ACCURACY_ONLY=True. |
| src/inference_endpoint/dataset_manager/dataset.py | Introduces Dataset.ACCURACY_ONLY class flag. |
| src/inference_endpoint/dataset_manager/init.py | Exposes/imports SWEBench dataset. |
| src/inference_endpoint/config/schema.py | Adds swe_bench_scorer to ScorerMethod and injects concurrency → SWE-bench workers default. |
| src/inference_endpoint/commands/benchmark/execute.py | Runs scorer preflight(), prevents accuracy-only datasets as perf, skips endpoint phases for external scorers, and improves sample counting/reporting. |
| src/inference_endpoint/config/templates/online_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/offline_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/concurrency_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| examples/10_Agentic_Inference/swebench_template.yaml | Adds a SWE-bench/mini-swe-agent config template used by the external scorer. |
| examples/10_Agentic_Inference/swe_bench_accuracy.yaml | Adds a runnable example config for SWE-bench external accuracy evaluation. |
| examples/10_Agentic_Inference/README.md | Documents how to sync/run the SWE-bench accuracy subproject. |
| examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/accuracy/RUNBOOK.md | Adds an ops runbook for validating the SWE-bench accuracy pipeline. |
| examples/10_Agentic_Inference/accuracy/pyproject.toml | Adds isolated uv subproject deps for SWE-bench evaluation tooling. |
| tests/unit/evaluation/test_swe_bench_scorer.py | Adds unit coverage for scorer behavior, config patching, and preflight. |
| tests/unit/dataset_manager/test_swe_bench_dataset.py | Adds unit coverage for dataset registration, caching, and subset mapping. |
| tests/unit/commands/test_benchmark.py | Adds tests for concurrency→workers injection, ACCURACY_ONLY enforcement, preflight propagation, and phase skipping behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
from
8d66f43 to
ae04494
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #342 +/- ##
=======================================
Coverage ? 80.60%
=======================================
Files ? 130
Lines ? 17781
Branches ? 0
=======================================
Hits ? 14333
Misses ? 3448
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Gate SWE-bench accuracy setup and scoring by benchmark mode so performance-only runs do not trigger external accuracy work. Preserve self-contained scorer artifacts, pin SWE-bench uv subprocesses to the subproject venv, report external sample counts accurately, and wrap preflight timeouts as SetupError. Also guard malformed scoring configs and keep the targeted SWE-bench coverage added during review follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
from
da45a99 to
fe442f8
Compare
Comment on lines
+155
to
+167
| def _to_rel(p: str) -> str: | ||
| p = p.lstrip("/") | ||
| if p.startswith("testbed/"): | ||
| p = p[len("testbed/") :] | ||
| return p | ||
|
|
||
| files_str = " ".join(shlex.quote(f"/testbed/{_to_rel(f)}") for f in files) | ||
| actions.append( | ||
| { | ||
| "command": f"echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && git -C /testbed diff HEAD -- {files_str}", | ||
| "tool_call_id": tool_call_id, | ||
| } | ||
| ) |
Comment on lines
+176
to
+180
| bash_cmd = ( | ||
| f"awk 'NR>={start} && NR<={end} " | ||
| f'{{printf "%6d\\t%s\\n", NR, $0}}\' ' | ||
| f"{shlex.quote(path)}" | ||
| ) |
The workers extra sets the mini-swe-agent run's parallelism and defaults to target_concurrency for concurrency/agentic_inference load patterns. Document this in the example README and note it in the kimi config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment on lines
+382
to
+388
| perf_cls = Dataset.PREDEFINED.get(perf_cfg.name) | ||
| if perf_cls is not None and perf_cls.ACCURACY_ONLY: | ||
| raise InputValidationError( | ||
| f"Dataset '{perf_cfg.name}' is accuracy-only and cannot be used " | ||
| "as a performance dataset. Use a different dataset (e.g. 'random') " | ||
| "for the performance phase." | ||
| ) |
Comment on lines
+155
to
+167
| def _to_rel(p: str) -> str: | ||
| p = p.lstrip("/") | ||
| if p.startswith("testbed/"): | ||
| p = p[len("testbed/") :] | ||
| return p | ||
|
|
||
| files_str = " ".join(shlex.quote(f"/testbed/{_to_rel(f)}") for f in files) | ||
| actions.append( | ||
| { | ||
| "command": f"echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && git -C /testbed diff HEAD -- {files_str}", | ||
| "tool_call_id": tool_call_id, | ||
| } | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds SWE-bench accuracy evaluation to the agentic inference workflow (per #310). A new
SWEBenchdataset loadsSWE-bench_Verified/_Lite, andSWEBenchScorerdrives the eval by shelling out tomini-swe-agentand theswebenchharness in an isolated uv subproject, bypassing the built-in accuracy phase.Agent-run parallelism (
extras.workers) defaults to the load pattern'starget_concurrencywhen unset. Includes example configs, an optional Qwen tool-call patch, and docs.Type of change
Related issues
Closes #310
Testing
Checklist