Skip to content

fix(evaluation): skip invocations without user events to prevent ValidationError#5916

Closed
Oxygen56 wants to merge 1 commit into
google:mainfrom
Oxygen56:fix/skip-invocations-without-user-events-3760
Closed

fix(evaluation): skip invocations without user events to prevent ValidationError#5916
Oxygen56 wants to merge 1 commit into
google:mainfrom
Oxygen56:fix/skip-invocations-without-user-events-3760

Conversation

@Oxygen56
Copy link
Copy Markdown

Description

Fixes #3760

EvaluationGenerator.convert_events_to_eval_invocations iterates over events grouped by invocation_id. For invocations that contain no user-authored event (e.g., internal background turns, system-driven tasks where all events have author != "user"), the code previously left user_content as an empty Content(parts=[]).

Invocation.user_content is typed as a required genai_types.Content. While Content(parts=[]) does not cause a Pydantic validation error, creating an Invocation with no meaningful user content is semantically incorrect — an invocation without a user turn is not meaningful for evaluation.

This fix: initializes user_content = None and skips the invocation if no user event is found. A DEBUG log line is emitted so operators can diagnose unexpected skips without producing noisy warnings.

Changes

  • src/google/adk/evaluation/evaluation_generator.py: initialize user_content = None instead of Content(parts=[]); skip invocations where no user event was found.

Testing

@google-cla
Copy link
Copy Markdown

google-cla Bot commented May 31, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label May 31, 2026
@adk-bot
Copy link
Copy Markdown
Collaborator

adk-bot commented May 31, 2026

Response from ADK Triaging Agent

Hello @Oxygen56, thank you for contributing to the adk-python project and submitting this pull request!

To help us review your PR more efficiently, please ensure it complies with our contribution guidelines:

  1. Contributor License Agreement (CLA): It looks like the CLA status check is currently failing. Please make sure to sign Google's CLA at https://cla.developers.google.com/ to proceed.
  2. Logs / Verification Screenshots: Since this is a bug fix, could you please provide a log sample or screenshot showing how the fix works after application? This helps reviewers better understand the impact.
  3. Unit Tests & Test Summary: Please write or update unit tests under tests/unittests/ to cover this new edge case (skipping invocations without user-authored events). In addition, please run the tests and provide a brief summary of the passed pytest results.

Thank you so much for your effort and support!

@Oxygen56 Oxygen56 force-pushed the fix/skip-invocations-without-user-events-3760 branch 3 times, most recently from 7265e58 to face828 Compare May 31, 2026 13:16
@Oxygen56
Copy link
Copy Markdown
Author

Oxygen56 commented Jun 1, 2026

Fixed the commit author to a GitHub-linked email. Triggering CLA recheck.

…dationError

EvaluationGenerator.convert_events_to_eval_invocations previously
initialized user_content as Content(parts=[]) for invocations without
user-authored events. While this is a valid Content object, creating
an Invocation with no meaningful user content is semantically incorrect
and can cause issues downstream in the evaluation pipeline.

This fix initializes user_content = None and skips the invocation if
no user event is found, emitting a debug log message for diagnostics.

Fixes google#3760

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Oxygen56 Oxygen56 closed this Jun 1, 2026
@Oxygen56 Oxygen56 force-pushed the fix/skip-invocations-without-user-events-3760 branch from face828 to aa3bb97 Compare June 1, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluation Invocation.user_content ValidationError when session has invocations without user events

2 participants