fix(evaluation): skip invocations without user events to prevent ValidationError#5916
Closed
Oxygen56 wants to merge 1 commit into
Closed
fix(evaluation): skip invocations without user events to prevent ValidationError#5916Oxygen56 wants to merge 1 commit into
Oxygen56 wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Collaborator
|
Response from ADK Triaging Agent Hello @Oxygen56, thank you for contributing to the To help us review your PR more efficiently, please ensure it complies with our contribution guidelines:
Thank you so much for your effort and support! |
7265e58 to
face828
Compare
Author
|
Fixed the commit author to a GitHub-linked email. Triggering CLA recheck. |
…dationError EvaluationGenerator.convert_events_to_eval_invocations previously initialized user_content as Content(parts=[]) for invocations without user-authored events. While this is a valid Content object, creating an Invocation with no meaningful user content is semantically incorrect and can cause issues downstream in the evaluation pipeline. This fix initializes user_content = None and skips the invocation if no user event is found, emitting a debug log message for diagnostics. Fixes google#3760 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
face828 to
aa3bb97
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3760
EvaluationGenerator.convert_events_to_eval_invocationsiterates over events grouped byinvocation_id. For invocations that contain no user-authored event (e.g., internal background turns, system-driven tasks where all events haveauthor != "user"), the code previously leftuser_contentas an emptyContent(parts=[]).Invocation.user_contentis typed as a requiredgenai_types.Content. WhileContent(parts=[])does not cause a Pydantic validation error, creating anInvocationwith no meaningful user content is semantically incorrect — an invocation without a user turn is not meaningful for evaluation.This fix: initializes
user_content = Noneand skips the invocation if no user event is found. ADEBUGlog line is emitted so operators can diagnose unexpected skips without producing noisy warnings.Changes
src/google/adk/evaluation/evaluation_generator.py: initializeuser_content = Noneinstead ofContent(parts=[]); skip invocations where no user event was found.Testing
test_evaluation_generator.pytests pass (all existing test cases include user events, so they are unaffected).Invocation.user_contentValidationError when session has invocations without user events #3760: sessions with invocations that have no user-authored event no longer produce invalid eval invocations.