chore(deps): update dependency promptfoo to v0.121.15

[renovate]

ℹ️ Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Change	Age	Confidence
promptfoo (source)	0.121.2 → 0.121.15

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

Release Notes

promptfoo/promptfoo (promptfoo)

v0.121.15

Compare Source

Features

• assertions: grade multimodal outputs (#​9617) (ec4d3b4)
• build: add DAG-aware architecture cycle checks (#​9557) (9d7d810)
• providers: add Azure MAI image provider and MAI model recognition (#​9595) (e091527)
• providers: upgrade MiniMax default model to M3 (#​9598) (43b30c5)

Bug Fixes

• auth: enable auto-share for on-prem Report Server after login (#​9612) (8ece593)
• auth: preserve cloud request headers (#​9610) (3a2c5e6)
• code-scan: process findings in mixed skip responses (#​9525) (aeae790)
• code-scan: reject fractional inline comment lines (#​9556) (0a2b329)
• eval: compose metadata and result filters (#​9604) (49539ee)
• eval: preserve --config ordering when expanding a directory entry (#​9606) (fc081a0)
• harden validation and target discovery (#​9614) (465d610)
• providers: require deployment name for no-default Azure provider types (#​9603) (da48abd)
• providers: suppress optional GCP metadata warning (#​9554) (8dcc44b)
• redteam: restrict remote auth to cloud host (#​9597) (1b2c6d9)

v0.121.14

Compare Source

Features

• add A2A provider (#​9586) (963b264)
• assertions: add agent-rubric grader (#​9453) (cadb3c5)
• build: publish a lightweight promptfoo/contracts subpath (#​9535) (6e89fd4)
• cli: add seeded random sampling (#​9522) (6e4ba60)
• eval: extract eval-creator readiness and validation modules (#​9397) (d3af118)
• eval: surface trace linkage on result rows (#​9027) (9468d66)
• providers: add OpenAI GPT-5.5/5.4 frontier models + Codex on Amazon Bedrock (#​9587) (1e35267)
• providers: add OpenAI originator header (#​9474) (6141070)
• providers: expose traceable agent turn markers (#​9475) (4018837)
• providers: promote Fireworks AI from registry stub to dedicated provider (#​9542) (2bae148)
• tracing: add runtime receiver controls (#​9028) (90df6a1)

Bug Fixes

• assertions: don't classify gen_ai.tool.definitions chat spans as tool calls (#​9524) (b59f397)
• cli: preflight force imports before collision lookup (#​9570) (0b93733)
• cloud: on-prem API host for guardrails and http-generator, with host-resolution tests (#​9580) (b9a014a)
• cloud: use on-prem API host in checkEmailStatus (#​9576) (063c62b)
• db: avoid SQLITE_LOCKED flakiness in shared-cache test database (#​9567) (1fdb59b)
• db: serialize libsql test database cleanup (#​9540) (f4380c2)
• deps: keep ModelAudit pydantic-core pinned to compatible 2.46.4 (b2b35b0)
• deps: update dependency ai to ^6.0.190 (#​9577) (9ec614a)
• eval: canonicalize retry JSONL output with atomic rewrites (#​9547) (8d7c920)
• eval: redact credentials from the persisted browser store (#​9396) (4d5bed5)
• evaluator: preserve and harden programmatic JSONL output (#​9538) (8ddd906)
• output: redact api-key and legacy transport headers in JSONL/DB (#​9546) (e194c85)
• providers: handle Codex SDK rate limits (#​9473) (76d3db4)
• providers: inject n8n sessions into custom bodies (#​9527) (9cc0542)
• providers: preserve n8n array body templates (#​9544) (6cdf63d)
• providers: preserve streamed Anthropic refusal guardrails (#​9560) (ff8eafd)
• providers: serialize persistent browser sessions (#​9414) (097ff9b)
• redteam: authenticate remote-generation requests against on-prem cloud (#​9584) (7df8fae)
• util: restore nested SAS tokens after array reorder (#​9528) (9759e5a)
• webui: render negative-only metric charts (#​9526) (7ccafa4)

v0.121.13

Compare Source

Features

• add MiniMax provider (#​8207) (ebd78f6)
• anthropic: add support for Claude Opus 4.8 (#​9500) (e090730)
• assertions: add ignore option to trajectory:tool-args-match (#​9466) (6423f2a)
• assertions: add metadata shortcut to assertion context (#​6863) (5264680)
• assertions: support glob patterns in trajectory:tool-args-match ignore (#​9484) (18b3eef)
• assertions: tolerate optional defaults in trajectory:tool-args-match (#​9471) (a74894a)
• cli: add prompt optimization (#​9294) (27d6dbe)
• cli: warn on deprecated Node.js 20 runtime (#​9494) (8de3408)
• eval: add eval runtime vars (#​9331) (3c76045)
• eval: add output detail deep links (#​9170) (1f5845b)
• eval: load test sets from azure blob storage (#​9282) (b5ea8ab)
• google: add Gemini 3.5 Flash and refresh Google/Vertex models (#​9376) (6c81c8d)
• output: add html report detail drawer (#​9291) (8c54598)
• output: add named metric columns to CSV exports (#​9480) (4531750)
• providers: add MLflow AI Gateway provider (#​8860) (d0e7e2d)
• providers: add n8n webhook provider (#​6770) (3dc6e32)
• providers: add Novita provider (#​8190) (42930e8)
• providers: add NVIDIA NIM provider (#​9491) (fa3e4a9)
• providers: add OrcaRouter provider (#​9493) (fe115ab)
• redteam: add runtime tags to redteam run (#​9499) (72d4cb4)

Bug Fixes

• app: clarify metric pill filtering (#​9252) (05e5a30)
• app: reorder eval output actions (#​9261) (91dc814)
• cache: make the fetch cache key stable across processes (#​9509) (7be3ef5)
• ci: avoid shell interpolation in telemetry check (#​9280) (4160a90)
• ci: disable deploy dependency cache (#​9281) (f3a2502)
• ci: pin code-scan Node to 24.15.0 to stop install timeout (#​9374) (e200bb5)
• ci: scope code-scan-action mirror sync to release artifacts (#​9363) (76c3a29)
• classify script stderr by severity (#​9459) (1eeef19)
• cli: validate optimization split input (#​9514) (e7e29d1)
• code-scan: emit structured fork PR skip output (#​9426) (61c624c)
• code-scan: honor minimum-severity alias when min-severity is unset (#​9433) (ea5ea9e)
• code-scan: suppress SARIF on skipped fork scans (#​9451) (8236e65)
• config: dedupe repeated executable providers (#​9430) (6325653)
• config: preserve all function providers in combineConfigs (#​9408) (ed4e29e)
• db: isolate libsql test databases (#​9504) (b01b257)
• db: parameterize JSON path filters (#​9345) (dc5e2ed)
• db: replace better-sqlite3 with libsql (#​9486) (ca3dea4)
• db: restore SQLite busy_timeout for the libsql driver (#​9508) (a74b2fe)
• deps: avoid Socket-blocked package releases (#​9456) (89d6e61)
• deps: bump brace-expansion and webpack-dev-server [security] (#​9339) (bb1d57e)
• deps: lazy-load optional integration packages (#​9284) (db37941)
• deps: patch qs denial of service advisory (#​9384) (b392051)
• deps: update anthropic packages (#​9461) (87277b9)
• deps: update dependency engine.io to v6.6.7 (#​9457) (5f36395)
• deps: update dependency engine.io to v6.6.8 (#​9477) (3366b67)
• deps: update dependency engine.io-client to v6.6.5 (#​9478) (32610e3)
• eval: accept defaultTest-only configs in prompt optimizer (#​9365) (f8ecebd)
• eval: error when optimizer pair has no runnable tests (#​9372) (45046ac)
• eval: handle Google Sheets export classification (#​9385) (5ffeb33)
• eval: invalidate count cache after retry cleanup (#​9431) (3ea3599)
• eval: invalidate history cache after result update (#​9487) (2e48494)
• eval: invalidate result-count cache on insert (#​9348) (#​9421) (4651ceb)
• eval: keep eval runtime vars out of persisted results (#​9370) (734db7c)
• eval: preserve all distinct CallApiFunction providers in combineConfigs (#​9402) (b7f2a93)
• eval: preserve optimized prompt routing (#​9391) (fff7ca4)
• eval: preserve prompt optimizer progress when a later round fails (#​9505) (211fadf)
• eval: prevent optimize from polluting jsonl output (#​9364) (1205a2a)
• eval: redact Azure Blob SAS tokens in outputs (#​9386) (23a5376)
• eval: refresh history cache after inserts (#​9496) (ed17310)
• eval: restore result header borders and live metrics (#​9273) (4ceaeb5)
• eval: reuse configured grading provider references (#​9460) (28b26d9)
• eval: show test descriptions in HTML output (#​9390) (206f1dd)
• handle chart and Google response edge cases (#​9409) (09701a8)
• http: isolate auth token cache (#​8636) (8d745cd)
• http: redact debug request metadata (#​8635) (6594491)
• mcp: use cryptographic session identifiers (#​9392) (c6f5144)
• providers: align Gemini safety block handling (#​9387) (45fed35)
• providers: harden callback file URL parsing (#​9479) (d6bfc1f)
• providers: honor Vertex default host overrides (#​9389) (11a8d9b)
• providers: improve invalid provider diagnostics (#​9405) (04c5aa4)
• providers: normalize Claude Opus compatibility (#​9510) (14e9dc1)
• providers: preserve AI Studio grounding metadata (#​9436) (79b6b63)
• providers: preserve raw Google stream chunks (#​9437) (2875f4e)
• providers: price unknown Grok Imagine slugs at the quality tier (#​9337) (efa689e)
• providers: redact MLflow Gateway api key and restore Novita routing test (#​9511) (9e2eaf5)
• providers: reject MiniMax function_call (#​9512) (b9a849b)
• providers: return refreshed HTTP auth tokens safely (#​9394) (9a258a5)
• providers: show labels in missing API key diagnostics (#​9404) (b43625b)
• providers: stop double-counting xAI reasoning tokens in cost (#​9326) (26f6d13)
• providers: stop double-counting xAI reasoning tokens in cost (#​9507) (7f2932c)
• secure script temporary artifacts (#​9393) (c320916)
• server: publish eval job completion only after results are saved (#​9395) (a9f5bcf)
• server: restore Azure Blob SAS tokens by value, not array position (#​9516) (b7f4a04)
• util: don't misreport a missing transitive dep as the optional package (#​9513) (e93c817)
• web-viewer: preserve config variable order instead of alphabetizing (#​9435) (a53d6f7)
• webui: avoid NaN in named metric charts (#​9352) (a6b860f)
• webui: clear row hint on eval selection (#​9380) (4586da4)
• webui: clear stale eval detail row hint (#​9377) (c9a3f61)
• webui: clear stale rowId and details hash on filter clear (#​9369) (e9b8e98)
• webui: harden eval table deep links and header borders (#​9285) (3514f6d)
• webui: make full metric chip area clickable (#​9368) (8eed102)
• webui: page eval deep links by rowId when filtered (#​9371) (16ede34)

Performance Improvements

• webui: speed up evaluation selection (#​9462) (dda3db3)

v0.121.12

Compare Source

Features

• cache: support repeat-aware fetch cache options (#​6844) (c119832)
• cli: add searchable team selector (#​9243) (247eef5)
• cli: import OpenAI Evals dashboard exports (#​9305) (70cebab)
• code-scan: add SARIF output support (#​9161) (4da26e9)
• code-scan: refine SARIF output ergonomics (#​9159) (ea3a655)
• eval: add metrics pills display toggle (#​9253) (eb7ecad)
• eval: support runtime tags (#​9322) (5a09980)
• examples: expand google adk integration (#​9047) (f190033)
• mcp: add response transforms (#​8943) (5d90bb7)
• providers: normalize OpenCode skill usage (#​9250) (8930dad)
• providers: support Agents SDK 0.9 workflows (#​9128) (0505ea1)
• redteam: support context purpose overrides (#​9260) (ca62ec9)
• telemetry: capture node runtime metadata (#​9206) (ec9d308)

Bug Fixes

• app: align table settings info buttons (#​9249) (a3252e9)
• app: improve custom metrics dialog readability (#​9248) (a3eafa4)
• app: sort custom metrics before truncation (#​9246) (74d2f9d)
• cache: include status in JSON parse errors (#​9317) (bb64b2e)
• cli: clearer errors for malformed eval imports (#​9333) (869fd54)
• code-scan: isolate action OIDC token env (#​9309) (30c99e9)
• code-scan: route logs to stderr for structured output (#​9329) (06cce72)
• code-scan: scope OIDC token to scan subprocess (#​9308) (178b57a)
• code-scan: skip unrelated default config startup (#​9230) (335d809)
• deps: avoid incompatible npm release-age config (#​9244) (b8bfb73)
• deps: bump nested ws to a non-vulnerable version (#​9330) (1fe31a4)
• deps: clear protobuf and langsmith alerts (#​9219) (d0995b8)
• deps: remove stray smithy lockfile root dependency (#​9271) (8f34641)
• deps: resolve active advisories (#​9153) (47cd609)
• deps: update anthropic packages (#​9187) (a68ed8b)
• deps: update dependency @​anthropic-ai/sdk to ^0.97.1 (#​9316) (88bc094)
• deps: update dependency @​inquirer/search to v4.1.9 (#​9277) (7a7ae06)
• deps: update dependency @​opentelemetry/exporter-trace-otlp-http to ^0.217.0 (#​9193) (082cb22)
• deps: update dependency ws to v8.20.1 [security] (#​9275) (bac18de)
• deps: update openai packages (#​9189) (f2ceff3)
• dev: use tsx watch for server (#​9301) (f0280e6)
• eval: allow header horizontal scrolling (#​9245) (4e3fb27)
• eval: preserve eval export import parity (#​9310) (ef7f98e)
• fetch: decompress fallback for compressed responses on Node 26 (#​9218) (93f5f1e)
• fetch: decompress pooled undici responses (#​9234) (e193d61)
• guard against empty choices and message=None in swe_runner example (#​9263) (b6ce944)
• isolate loadFunction cache entries (#​9255) (1e3533e)
• mcp: surface tool error results (#​9320) (b600a27)
• openclaw: support protocol negotiation for versions 3 and 4 (#​9254) (4220562)
• providers: aggregate command output deltas (#​9204) (5f89b7c)
• providers: avoid constant-key cache HMAC (#​9152) (2b5db89)
• providers: decompress mTLS HTTP responses on Node 26 (#​9325) (4ff8315)
• providers: execute Anthropic MCP tool results (#​9286) (a42dbd6)
• providers: harden codex app-server approvals and raw evidence (#​9131) (2299c31)
• providers: harden OpenAI Realtime provider after #​9137 (#​9156) (9447d18)
• providers: honor max_tool_calls: 0 for Anthropic MCP (#​9327) (5b804e0)
• providers: map Python worker stderr log levels (#​9306) (bbd683c)
• providers: preserve interleaved command output deltas (#​9210) (3c03fbe)
• providers: refresh xai grok support (#​9237) (58b1c6e)
• providers: support realtime tools in persistent sessions (#​9137) (00f6518)
• providers: tear down Realtime socket on response timeout (#​9324) (d3bb58c)
• redteam: clarify empty custom intent scans (#​8990) (93b5579)
• redteam: improve setup control accessibility (#​9188) (0ea2f86)
• redteam: redact plugin config logs (#​8661) (5ae731b)
• resolve code quality findings (#​9151) (83d80b8)
• server: cap eval table page size (#​9313) (25286aa)
• ui: improve table filter accessibility (#​9166) (cb5dab7)
• ui: wrap static table filters (#​9167) (9fffdca)
• webui: align custom intent count gating (#​9201) (007ae19)
• webui: anchor nav menus to their triggers (#​9154) (cfb474d)
• webui: improve data table keyboard access (#​9171) (c4ce1b4)
• webui: improve eval results table sizing (#​9257) (5269f22)
• webui: restore nav dropdown animations and clean up after #​9154 (#​9169) (d0337d7)

v0.121.11

Compare Source

Features

• quiverai: add Arrow 1.1 models, vectorize endpoint, and GPT Image-2 pipeline (#​9139) (ce2c62d)

Bug Fixes

• redteam: handle MCP target prompt materialization (#​9149) (a050023)
• redteam: keep fresh target type state in sync (#​9145) (770130f)
• redteam: show target type selector immediately (#​9141) (c143fa6)

v0.121.10

Compare Source

Features

• app: humanize per-cell latency in eval results table (#​8998) (509aa54)
• bedrock: add converse mcp support (#​9134) (c1fe183)
• claude-agent-sdk: bump SDK to 0.2.120 and expose new options (#​8946) (350a24c)
• cli: add eval filter range (#​8895) (f049dd3)
• examples: add integration-inspect-osworld wrapper (#​8932) (5d3279a)
• openapi: generate server spec from route DTOs (#​8928) (c92102b)
• output: add junit xml reports (#​9073) (ec8149c)
• providers: add Atlas Cloud provider (#​8980) (90da094)
• providers: add first-class n parameter support for OpenAI chat completions (#​8914) (6e120b1)
• providers: expose OpenAI prompt cache controls (#​9093) (9ec9c1f)
• providers: update mistral support (#​9018) (130f59f)
• xai: refresh provider support (#​9033) (c2c7abf)

Bug Fixes

• api: add DTO validation for core server routes (#​8922) (87f637f)
• api: correct user-visible response regressions (#​8976) (74e7f95)
• api: harden node package contracts (#​9055) (be44d4e)
• api: make modelAudit safeRespond fire-safe under parse failures (#​8977) (524c5e6)
• api: preserve error details and stack traces in shared helpers (#​8975) (ed0b67b)
• api: validate eval route DTOs (#​8924) (84d5c8c)
• api: validate media, blob, user, and trace DTOs ([#&#8203

✂ Note

PR body was truncated to here.

---

Configuration

📅 Schedule: (in timezone Europe/London)

• Branch creation
  • "before 10am on friday"
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[github-actions]

Caution

[High Risk] New API EC2 instance will be exposed as a direct public endpoint with a host-level HTTP service on port 9090

This change creates a new EC2 API host and associates Elastic IP 13.134.236.98 to it, while also attaching the instance to the API health target group on port 9090. In the current environment, the same pattern already produces a directly reachable instance: 540044833068.eu-west-2.ec2-instance.i-025efedc46bef3be1 has a public EIP, sits behind customer-facing and internal security groups, and is reachable on ports used for HTTPS and health monitoring. The new instance’s user data starts an HTTP server bound to 0.0.0.0:9090, so the replacement will expose a host-level health endpoint to broad internal ranges and make the service reachable as a direct EC2 endpoint rather than only through managed edge controls.

That violates the org requirement that EC2 instances must not be directly internet reachable and creates a real segmentation regression: customer whitelisted traffic and internal 10.0.0.0/8 traffic can hit the instance itself, not just a managed proxy layer. If the same shared security-group pattern is applied to the new instance as it is to the existing one, this will widen lateral movement paths internally and expose the port 9090 health service on a production API host.
_{View reasoning tree here.}

Warning

[Medium Risk] New production EC2 instance is launched without an IAM role and with direct public exposure

The new production EC2 instance 540044833068.eu-west-2.ec2-instance.i-025efedc46bef3be1 is being created as a standalone public-facing host with an Elastic IP and no attached IAM instance profile. The blast-radius state for the corresponding live instance shows IamInstanceProfile: null, a public IP of 18.133.94.23, and direct attachment to customer-facing and internal security groups. This violates the organization's EC2 access-control and network-access baselines and creates a concrete risk that the server will either rely on hardcoded credentials for AWS access or fail when it needs to call AWS APIs, while also being directly reachable from the internet in production.

The hypothesis's ARM/x86 boot-failure path is not supported here: the live instance is already running on t4g.nano with Architecture: arm64, and its root volume and instance status are healthy. But the same change still introduces a real operational and security risk through missing machine identity and public exposure. If this host is used for production API access without a least-privilege role, deployments, secret retrieval, logging, or future integrations will fail or encourage unsafe credential handling, and the public endpoint increases attack surface against a production instance.
_{View reasoning tree here.}

Signals

Routine → Multiple compute and access resources are showing unusual infrequent updates at 1 event/week for the last 3 months, with several related resources changing only 2 events/week for the last 3 months.
Policies → Infrastructure resources showing unusual policy violations that may need review: the S3 bucket is missing required tags and does not have server-side encryption configured, while the security group allows SSH (port 22) access from anywhere (0.0.0.0/0).

_{Additional Change Details:} Items 106 Edges 194 model|risks_v6 ✨Encryption Key State Risk ✨KMS Key Creation

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Auto-blocked: Routine score (-5) is below minimum (-1)

---

📊 Signals Summary

Routine 🔴 -5

---

🔥 Risks Summary

High 0 · Medium 0 · Low 0

---

💥 Blast Radius

Items 23 · Edges 75

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Auto-blocked: Policy signal (-3) is below threshold (-2); Routine score (-5) is below minimum (-1)

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 0 · Medium 0 · Low 0

---

💥 Blast Radius

Items 5 · Edges 20

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Found 2 high risks requiring review

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 2 · Medium 0 · Low 0

---

💥 Blast Radius

Items 107 · Edges 219

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Found 1 high risk requiring review

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 1 · Medium 1 · Low 0

---

💥 Blast Radius

Items 63 · Edges 135

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Found 2 high risks requiring review

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 2 · Medium 0 · Low 0

---

💥 Blast Radius

Items 93 · Edges 217

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Auto-blocked: Policy signal (-3) is below threshold (-2); Routine score (-5) is below minimum (-1)

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 0 · Medium 0 · Low 0

---

💥 Blast Radius

Items 79 · Edges 192

---

View full analysis in Overmind ↗