Feature/lld ir#449
Conversation
- add relevance, effectiveness, and authority evaluators - add standalone and combined executor-based evaluation scripts - support query-level and result-level classified outputs - improve LLM response parsing and content issue detection - set overall weights to 0.7/0.2/0.1 - add evaluator tests and usage documentation
- add relevance, effectiveness, and authority evaluators - add standalone and combined executor-based evaluation scripts - support query-level and result-level classified outputs - improve LLM response parsing and content issue detection - set overall weights to 0.7/0.2/0.1 - add evaluator tests and usage documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive search result quality evaluation metrics (relevance, effectiveness, and authority) using Dingo's LocalExecutor, along with documentation, example scripts, and unit tests. The review feedback highlights several robust improvements to prevent runtime errors and parsing failures: updating regex patterns in the relevance grader to correctly handle the reasoning field when it is the last key in a JSON object, explicitly verifying that locations is a list before iterating over it, and replacing a risky and inefficient json.dumps call in the authority grader with a safer iteration-based check for DOI strings.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
- add relevance, effectiveness, and authority evaluators - add standalone and combined executor-based evaluation scripts - support query-level and result-level classified outputs - improve LLM response parsing and content issue detection - set overall weights to 0.7/0.2/0.1 - add evaluator tests and usage documentation
# Conflicts: # dingo/model/llm/llm_search_result_authority.py # examples/retrieval/sdk_eval_authority.py # examples/retrieval/sdk_eval_effectiveness.py # examples/retrieval/sdk_eval_relevancy.py # examples/retrieval/sdk_eval_search_result.py
2 participants
feat(retrieval): add search result quality evaluation