add Verifiers v1 support to Prime CLI#758
Conversation
xeophon
changed the title
xeophon
requested review from
JannikSt,
JohannesHa,
burnpiro,
d42me,
kcoopermiller and
willccbb
as code owners
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 518d3d9d3a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a3d41abef6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
mikasenghaas
left a comment
mikasenghaas
left a comment
There was a problem hiding this comment.
we get a weird merge of typer cli (prime cli) and pydantic config (vf/ prl). imo broader question if we want to adopt pydantic config everywhere. would prob need some changes for look + feel, and subcommands to work etc. maybe a nicer fusion is possible tho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2499cbbbc9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Pass `@ config.toml` to the v1 CLI as two tokens; the pinned prime-pydantic-config only reads a root config file in that form, so the single-token `@config.toml` failed with "Unrecognized arguments". - Local v1 evals no longer probe hub owners or install remote envs prime-side: references (bare names, org/name[@Version]) pass through verbatim and verifiers resolves them (`ensure_installed`). Prime only pre-installs local ./environments checkouts and keeps platform metadata for the results upload. Hosted evals and the v0 --save-results path keep the existing resolution (v0 does not self-install). - Config peek accepts both `taskset.id` and the legacy top-level `id` (verifiers' EvalConfig.is_legacy rule), so v1-format configs run v0 envs through the legacy bridge. - Declare prime-pydantic-config[toml]>=0.3.0.dev86 directly, matching the verifiers constraint.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b9f799232
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Resume: only reuse the X-Prime-Eval-Env-Display header as the upload slug when it has slug shape; display values like "name (local - ahead of org/name)" contain a slash but are not slugs. - Results conversion: use vf's WireTrace alias; split prompt/completion at the first assistant turn; only the final branch carries the trace reward (earlier branches have no per-step reward, matching v0). - Fail fast with a pointed message when an eval config is in the old v0 format (env_id / [[eval]]) instead of surfacing a v1 parse error.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f696288f9b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- A bare --dry-run (no value or followed by another flag) now skips the inference preflight, matching the v1 parser's bool-flag semantics. - The --save-results path now skips the automatic upload when the local checkout is ahead of its tracked upstream, mirroring the v1 path, so unpublished local code is never attributed to the stale upstream env.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 56c29ebd34
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…l rename Hosted evals: - prime eval --hosted <v1 config.toml> (top-level [taskset]) submits the platform's raw `environments` payload; published envs keep environment_ids - hosted CLI flags collapse to one explicit-fields parse (no double argparse) Sandbox / v0 compatibility (verifiers no longer ships the v0 evaluator): - the frozen v0 argv surface (--save-results marker) and transitional [[eval]] TOMLs (incl. the platform's group_size dialect) convert onto the v1 CLI via verifiers.v1.cli.eval.compat; v0 env ids ride the legacy auto-detect, hub-slug taskset ids are installed and pinned to local names - --disable-tui/--debug map to --rich false; --disable-tui stays in the help so the platform's TUI probe keeps matching Command surface: - prime eval <env> / @ config.toml is the canonical command; bare and --help invocations show the eval help with subcommands appended; `prime eval run` remains a hidden alias for hosted-eval sandboxes
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bd55bbc26e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The platform's script generator moves to v1-native invocations (pinned toolchain), so prime no longer parses or converts v0 argv: - utils/verifiers_v0_cli.py deleted; transitional TOMLs convert inside verifiers at its ingestion boundary, v0 env ids ride the legacy auto-detect, and hub-slug ids in v1 configs are installed and pinned into a config copy (_pin_config_env) - hosted override flags speak the shared v1 dialect, parsed by verifiers' own EvalConfig (-n/-r/-m/--sampling.* and their aliases); endpoint/env payloads live in the config file - --save-results/--disable-tui/--debug/--sampling-args and the other v0-only flags are gone; unknown flags fail the v1 parse with a clean error
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4f63efe836
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- a flag-like --resume "value" falls through to the normal invocation so verifiers reports the missing directory instead of prime treating the flag as an output dir - config-driven validate pins hub env refs into a local config copy via _pin_config_env, matching the eval path
|
Re the typer/pydantic-config fusion (review from Jun 23): partially converged since. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7a812a7. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a812a7c3d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if _is_config_target(environment): | ||
| hosted_target_configs = _load_hosted_eval_configs(environment) | ||
| v1_target = _load_v1_hosted_target(Path(environment)) |
There was a problem hiding this comment.
When a hosted config is passed in the compact root-config form, e.g. prime eval @configs/eval.toml --hosted, the earlier normalization only handles a standalone @, so environment still includes the leading @ here. _is_config_target() accepts it because it ends in .toml, but _load_v1_hosted_target(Path(environment)) then tries to open the literal @configs/eval.toml path and the hosted run fails even though the same config works with @ configs/eval.toml and on the local path.
Useful? React with 👍 / 👎.
…ides - _prime_runtime_env injects PRIME_API_KEY / PRIME_INFERENCE_URL / PRIME_TEAM_ID into every verifiers subprocess now that verifiers reads env vars only; an explicit --api-key-var removes the injected key - hosted override parsing accepts legacy --header pairs alongside v1 --client.headers and folds both into the submission payload - import build_extra_headers from its v1 home (verifiers.v1.cli.eval.compat)
There was a problem hiding this comment.
💡 Codex Review
Preserve --env-dir-path for hosted runs
When a direct hosted run points at a local checkout outside the default, e.g. prime eval my-env --hosted --env-dir-path /tmp/envs, this variable is never populated from passthrough_args; the target built below therefore passes None into _resolve_hosted_environment, which falls back to ./environments. That makes the resolver miss the selected checkout/metadata and either fail the hosted run or attach it to a stale default checkout. Please parse/pop --env-dir-path before hosted override parsing and forward it to the target config.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| environment = {key: raw[key] for key in _V1_HOSTED_ENV_KEYS if raw.get(key) is not None} | ||
| optional = { | ||
| "num_examples": _first_config_value(raw, "num_tasks", "num_examples"), |
There was a problem hiding this comment.
When a raw v1 hosted TOML contains [taskset] but omits num_tasks, local v1 semantics mean all tasks (the CLI mapping above converts an explicit unset to -1), but this loader leaves num_examples absent so the later hosted default of 5 is used. A config like [taskset]\nid = "gsm8k-v1" will silently run only five tasks on hosted instead of the full taskset; set num_examples to the platform's all-tasks sentinel for v1 configs when the field is omitted.
Useful? React with 👍 / 👎.
build_extra_headers now takes (headers, header) directly, merge_sampling_args dropped its prefer_existing_keys knob, and PROVIDER_CONFIGS moved from the deleted verifiers.scripts.eval to verifiers.v1.cli.eval.compat (the guarded provider-URL fallback had been silently dead since that module was removed).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 613a9e0cf4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if "headers" in parsed.client.model_fields_set: | ||
| default_headers = type(parsed.client)().headers | ||
| client_headers = { | ||
| key: value | ||
| for key, value in parsed.client.headers.items() | ||
| if key not in default_headers or default_headers[key] != value | ||
| } |
There was a problem hiding this comment.
When a hosted run uses v1 client flags such as prime eval gsm8k --hosted --client.base-url https://api.openai.com/v1 --client.api-key-var OPENAI_API_KEY, pydantic_cli(EvalConfig, ...) parses those values under parsed.client, but this block only copies client.headers into overrides. The later hosted payload only gets api_base_url/api_key_var from overrides or TOML, so the CLI form silently runs with the default Prime inference settings instead of the requested endpoint/key.
Useful? React with 👍 / 👎.
| hosted_target_configs = [ | ||
| { | ||
| "env_id": environment, | ||
| "env_dir_path": env_dir_path, | ||
| "model": DEFAULT_MODEL, | ||
| "num_examples": HOSTED_RUN_DEFAULT_NUM_EXAMPLES, | ||
| "rollouts_per_example": HOSTED_RUN_DEFAULT_ROLLOUTS_PER_EXAMPLE, | ||
| "env_args": None, | ||
| "timeout_minutes": None, | ||
| "allow_sandbox_access": True, | ||
| "allow_instances_access": False, | ||
| "allow_tunnel_access": True, | ||
| "sampling_args": None, | ||
| "max_concurrent": None, | ||
| "max_retries": None, | ||
| "state_columns": None, | ||
| "independent_scoring": False, | ||
| "verbose": False, | ||
| "headers": None, | ||
| "extra_env_kwargs": None, | ||
| "api_client_type": None, | ||
| "api_base_url": None, | ||
| "api_key_var": None, | ||
| "eval_name": None, | ||
| } | ||
| {"env_id": environment, "env_dir_path": env_dir_path, "model": DEFAULT_MODEL} | ||
| ] |
There was a problem hiding this comment.
For direct hosted evals with a local checkout outside the default directory, e.g. prime eval my-env --hosted --env-dir-path /tmp/envs, this branch never populates env_dir_path from the CLI before building hosted_target_configs, so the resolver is given None and falls back to ./environments (or the Prime-only flag is rejected by the v1 parser). That prevents hosted resolution from finding the selected local metadata/upstream and can incorrectly reject a published local env; pop and assign --env-dir-path before calling the hosted override parser/building the target.
Useful? React with 👍 / 👎.
2 participants
Overview
Adds Verifiers v1 support to the Prime CLI while preserving compatibility with legacy and hosted evaluation flows. Verifiers is the source of truth for environment and config resolution; the Prime CLI's own machinery is scoped to hosted evaluations and results upload.
What changed
prime eval runandprime env initthrough the Verifiers v1 entrypoints@ config.tomlto the v1 CLI as two tokens — the only root-config form prime-pydantic-config parses — so bothprime eval @ config.tomlandprime eval @config.tomlworkorg/name[@version]references pass through verbatim (hub ids install on demand viaensure_installed); prime no longer probes hub owners or installs remote envs on the v1 path, and only pre-installs local./environmentscheckouts plus keeps platform metadata for uploads[taskset] id(v1 taskset) or top-levelid(legacy v0 env via the v1 bridge)--save-resultslegacy path on the existing v0 resolution (the v0 evaluator does not self-install hub environments)verifiers>=0.1.15.dev371from PyPI instead of a Git source, and declaresprime-pydantic-config[toml]>=0.3.0.dev86directly (the same constraint verifiers uses)exclude-newerpolicyuv syncanduv runworkflow for Python 3.11–3.13User impact
Local evaluations use the Verifiers v1 CLI by default and resolve environments exactly as
uv run evaldoes.prime eval @ config.tomlworks for both v1 tasksets and legacy v0 environments. Legacy and hosted-generated commands remain compatible with the existing hosted runner, while local v1 results upload through the Prime Evals flow. Once verifiers ships legacy env auto-detection (PrimeIntellect-ai/verifiers#1857),prime eval <v0-env>needs no flags at all.Note
High Risk
Large CLI/eval surface area changes (local, hosted, push, validate) with new dependency pins and dropped Python versions; regressions could break eval runs or platform uploads.
Overview
Local eval is now the default
prime eval <environment>path (withprime eval runkept as a hidden alias). It shells out to Verifiers v1 (verifiers.v1.cli.*) instead of the legacy eval module, using static command mapping and-centrypoint shims wherepython -mis unreliable. Config runs use@ pathfor pydantic-config; hub slugs in TOML are installed and pinned to local importable names before verifiers runs.The bridge adds
prime eval validate, resume /@config handling, v1--client.*defaults (headers, job id, output dir), dry-run skipping inference preflight, and post-run upload viaread_upload_data. Push and auto-upload delegate artifact reading to verifiers (nativeconfig.toml+results.jsonlor legacy metadata).Hosted
--hostedevals parse overrides throughEvalConfig/ v1 flags (-n,-r,--sampling.*,--client.headers) instead of argparse; v1 taskset TOMLs can submit rawenvironmentspayloads without a published hub env. Legacy[[eval]]/env_idconfigs still use the older hosted loader.Dependencies:
verifiers>=0.1.15.dev371andprime-pydantic-config[toml]from PyPI (Git pin removed);requires-pythonis>=3.11,<3.14. CI/release matrices drop 3.10/3.14 and align Docker on 3.11–3.13. Lab UI copy switches hosted eval commands toprime eval(norun).Reviewed by Cursor Bugbot for commit 613a9e0. Bugbot is set up for automated code reviews on this repo. Configure here.