Skip to content

fix(eval): handle failed inference results without invocations#5878

Open
he-yufeng wants to merge 1 commit into
google:mainfrom
he-yufeng:fix/eval-failed-inference-none
Open

fix(eval): handle failed inference results without invocations#5878
he-yufeng wants to merge 1 commit into
google:mainfrom
he-yufeng:fix/eval-failed-inference-none

Conversation

@he-yufeng
Copy link
Copy Markdown
Contributor

@he-yufeng he-yufeng commented May 28, 2026

What changed

  • Return a failed EvalCaseResult when inference failed before producing any invocations.
  • Preserve the existing session lookup path when a failed inference still has a session id.
  • Add a regression test for InferenceResult(status=FAILURE, inferences=None).
  • Clean up two existing lint issues in the touched eval test file so the local changed-file ruff check passes.

This prevents the eval runner from replacing the original inference error with TypeError: object of type 'NoneType' has no len().

Fixes #5876

To verify

  • .\.venv\Scripts\python.exe -m py_compile src\google\adk\evaluation\local_eval_service.py tests\unittests\evaluation\test_local_eval_service.py
  • .\.venv\Scripts\python.exe -m pytest tests\unittests\evaluation\test_local_eval_service.py -k "failed_without_inferences or evaluate_single_inference_result" -q --basetemp .tmp\pytest
  • .\.venv\Scripts\python.exe -m ruff check src\google\adk\evaluation\local_eval_service.py tests\unittests\evaluation\test_local_eval_service.py
  • .\.venv\Scripts\python.exe -m pyink --check src\google\adk\evaluation\local_eval_service.py tests\unittests\evaluation\test_local_eval_service.py
  • git diff --check

@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label May 28, 2026
@rohityan rohityan self-assigned this May 28, 2026
@rohityan rohityan added the request clarification [Status] The maintainer need clarification or more information from the author label May 28, 2026
@rohityan
Copy link
Copy Markdown
Collaborator

Hi @he-yufeng , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Please fix formatting errors by running autoformat.sh

@he-yufeng he-yufeng force-pushed the fix/eval-failed-inference-none branch from 7a33065 to 0315922 Compare May 28, 2026 20:51
@he-yufeng he-yufeng changed the title fix: handle failed eval inference results fix(eval): handle failed inference results without invocations May 28, 2026
@he-yufeng
Copy link
Copy Markdown
Contributor Author

Thanks for the review. I rebased onto the latest main, ran pyink on the touched files, and also cleaned two existing lint issues in the touched test file so the changed-file checks are clean. Local verification now passes for py_compile, the targeted eval tests, ruff, pyink --check, and git diff --check.

@he-yufeng
Copy link
Copy Markdown
Contributor Author

Rebased onto current upstream/main and reran the focused eval validation. Validation: test_local_eval_service.py passed (19 passed); ruff check passed; pyink --check passed; py_compile passed; git diff --check passed. I set PYTHONPATH=src so pytest imports this worktree explicitly.

@boyangsvl boyangsvl force-pushed the fix/eval-failed-inference-none branch from daddfd8 to 64759da Compare May 29, 2026 18:01
@rohityan rohityan added needs review [Status] The PR/issue is awaiting review from the maintainer and removed request clarification [Status] The maintainer need clarification or more information from the author labels May 29, 2026
@rohityan
Copy link
Copy Markdown
Collaborator

rohityan commented May 29, 2026

Hi @ankursharmas , can you please review this.

@rohityan rohityan requested a review from ankursharmas May 29, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation needs review [Status] The PR/issue is awaiting review from the maintainer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LocalEvalService crashes with len(None) when inference_result.inferences is None

3 participants