diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 421e6b3cf..5bbadc459 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -9,18 +9,172 @@ produced them. > routing, account state, network conditions, cache state, generated output, > and upstream behavior can materially change latency and token usage. -## Published result +## Published results -The first release-bound benchmark was recorded from the immutable Windows x64 -asset for `codex-raw 0.1.0`. One uninterrupted invocation completed exactly -two warm-up turns and twenty measured turns. It performed no retries and had -no failed or timed-out request. +Each row is bound to a named immutable release, exact binary digest, sanitized +raw samples, and a provenance record. The two releases were measured on the +same workstation with the same harness, account, model, prompt, and workload, +but on different dates. They are dated observations, not a controlled +before/after performance comparison. | Record ID | Release | Mode | Model | Measured runs | Median TTFT | p95 TTFT | Median total | p95 total | | --------------------------------------- | ------------------ | ------------ | -------------- | ------------: | ----------: | ----------: | -----------: | ----------: | +| `raw-0.1.1-app-windows-x64-20260729-01` | `codex-raw-v0.1.1` | `app-server` | `gpt-5.4-mini` | 10 | 815.77 ms | 3,022.92 ms | 983.73 ms | 3,292.14 ms | +| `raw-0.1.1-api-windows-x64-20260729-01` | `codex-raw-v0.1.1` | `api-server` | `gpt-5.4-mini` | 10 | 704.79 ms | 1,900.71 ms | 916.01 ms | 2,170.69 ms | | `raw-0.1.0-app-windows-x64-20260728-01` | `codex-raw-v0.1.0` | `app-server` | `gpt-5.4-mini` | 10 | 701.73 ms | 969.14 ms | 904.12 ms | 1,125.31 ms | | `raw-0.1.0-api-windows-x64-20260728-01` | `codex-raw-v0.1.0` | `api-server` | `gpt-5.4-mini` | 10 | 658.40 ms | 2,024.18 ms | 874.14 ms | 2,251.11 ms | +## Codex Raw 0.1.1 — Windows x64 + +This run used the immutable Windows x64 asset for `codex-raw 0.1.1`. One +uninterrupted invocation completed exactly two warm-up turns and twenty +measured turns. It performed no retries and had no failed or timed-out request. + +### 0.1.1 artifact identity + +| Field | Exact observed value | +| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | +| Release | [`codex-raw-v0.1.1`](https://github.com/bproject07/Codex-Source/releases/tag/codex-raw-v0.1.1), release ID `361516591`, published and immutable | +| Tag object | `42336d4b4f7710aa6ec64f41556263d64d0695de` | +| Tag commit | `4d814d6e7578a833bb355665862c0f4b83678f89` | +| Asset | `codex-raw-0.1.1-x86_64-pc-windows-msvc.zip`, 12,163,128 bytes | +| Asset SHA-256 | `f883916658fd6b4e1d549dbb50a532c72d9f9efafbdf7dfb7ee129740ff1057d` | +| Extracted binary SHA-256 | `0bb4a1d8b2d624d59b7023bc9256cb5643a5c5f9cf46f6b90bf21aeea6026e0c` | +| Version output | `codex-raw 0.1.1` | +| Target | `x86_64-pc-windows-msvc` | +| Benchmark harness | `codex-rs/raw-cli/scripts/benchmark_persistent.ps1` from the tagged commit | +| Harness Git blob | `2c92994556a50dd2c83be9c844ad41d141a6fbc8` | +| Harness blob SHA-256 | `95184d60b7bf753d97d1868263bdb369a3049b3a7f3d6d04127e81e0529e36a1` | +| Harness checkout SHA-256 | `a3b17cd9c9c3445bfbe07c530cb8884f3a24f755c772dcc4822368525edc16` (observed CRLF Windows checkout) | + +The downloaded archive digest matched both GitHub's asset digest and the +release's `SHA256SUMS` record before extraction. + +### 0.1.1 environment and workload + +| Field | Exact observed value | +| -------------------- | ------------------------------------------------------------------------------------- | +| Machine | VirtualBox virtual machine (`innotek GmbH`) | +| CPU exposed to guest | 12th Gen Intel Core i7-12700, 4 physical/logical guest cores | +| Memory | 12,776,771,584 bytes (about 11.90 GiB) | +| Operating system | Microsoft Windows 11 Pro, version/build `10.0.26200` | +| Architecture | x64 OS and x64 process | +| PowerShell | `7.6.3` | +| Local time zone | `FLE Standard Time` | +| Network | Local workstation connection; route and upstream model placement were not controlled | +| Start UTC | `2026-07-29T05:07:02.1434541Z` | +| End UTC | `2026-07-29T05:07:27.2418112Z` | + +The observed VM wall-clock interval is inconsistent with the monotonic +samples: finish minus start is 242.5499 ms shorter than the sum of the +sequential measured-turn durations alone. This can occur when a VM wall clock +is corrected during a run, but no cause is asserted here. The timestamps are +retained as observed metadata only. Every reported latency comes from the +harness's monotonic stopwatch and does not use the wall clock. + +The workload and calculation method were identical to the `0.1.0` run: +`gpt-5.4-mini`, public prompt `hello`, one app-server and one loopback +api-server, one warm-up per mode, ten measured turns per mode, sequential +requests, no retries, and a 120-second per-operation timeout. Reasoning effort +and service tier were omitted so the executable defaults applied. The prompt's +UTF-8 SHA-256 was +`2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824`. + +### 0.1.1 detailed results + +| Mode | Metric | Median | p90 | p95 | Minimum | Maximum | Mean | +| ------------ | ------ | ----------: | ----------: | ----------: | --------: | ----------: | ----------: | +| `app-server` | TTFT | 815.77 ms | 2,390.88 ms | 3,022.92 ms | 622.08 ms | 3,022.92 ms | 1,215.59 ms | +| `app-server` | Total | 983.73 ms | 2,588.05 ms | 3,292.14 ms | 826.19 ms | 3,292.14 ms | 1,419.76 ms | +| `api-server` | TTFT | 704.79 ms | 1,484.27 ms | 1,900.71 ms | 517.10 ms | 1,900.71 ms | 899.22 ms | +| `api-server` | Total | 916.01 ms | 1,693.17 ms | 2,170.69 ms | 722.06 ms | 2,170.69 ms | 1,114.33 ms | + +| Invariant or outcome | Observed value | +| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | +| Successful warm-up turns | 2 of 2 | +| Successful measured turns | 20 of 20 | +| Failed / timed-out turns | 0 / 0 | +| Input tokens | exactly 7 in every measured turn | +| Output lengths | 19 responses had 32 output characters; one API response had 29 output characters | +| Output text and fingerprints | deliberately not retained | +| Cached-input, output, and reasoning tokens | not retained by the harness; no values are inferred | + +With only ten samples per mode, nearest-rank p95 is the maximum observed +sample. Two app-server samples took about 2.39 and 3.02 seconds to first text, +which materially increased that mode's p90, p95, and mean. The API median was +110.97 ms lower for TTFT and 67.72 ms lower for total latency in this run, but +the small sample and uncontrolled upstream conditions do not establish a +general speed advantage or a release-to-release regression. + +### 0.1.1 published raw data + +| File | SHA-256 | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| [`benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.samples.json`](benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.samples.json) | `a56015e18430d0c220d0d2f04fb4d826b1d03c02ea47628fb9225ca2d607e0f5` | +| [`benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.provenance.json`](benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.provenance.json) | `52cc3e24478313f8bdf2d40d51aaba9ede70c1847641f00c18a3b4bbadc77860` | + +The samples file contains one sanitized row for every measured turn. It +contains order, mode, TTFT, total latency, input tokens, and output length. It +contains no response text or output fingerprint: unsalted hashes of +low-entropy outputs can reveal the text they are intended to hide. It also +contains no hostname, account identifier, credential, token, private address, +or absolute local path. + +### Reproduce the 0.1.1 aggregate run + +From a clone of this repository, download and verify the exact release asset, +then create a detached worktree for the exact tagged harness: + +```powershell +$archive = 'codex-raw-0.1.1-x86_64-pc-windows-msvc.zip' +$expected = 'f883916658fd6b4e1d549dbb50a532c72d9f9efafbdf7dfb7ee129740ff1057d' + +Invoke-WebRequest ` + "https://github.com/bproject07/Codex-Source/releases/download/codex-raw-v0.1.1/$archive" ` + -OutFile $archive +if ((Get-FileHash $archive -Algorithm SHA256).Hash.ToLowerInvariant() -ne $expected) { + throw 'release asset digest mismatch' +} + +Expand-Archive $archive -DestinationPath .\codex-raw-0.1.1 +$exe = Resolve-Path ` + '.\codex-raw-0.1.1\codex-raw-0.1.1-x86_64-pc-windows-msvc\codex-raw.exe' +& $exe --version + +git fetch origin tag codex-raw-v0.1.1 +git worktree add --detach .\codex-raw-0.1.1-source codex-raw-v0.1.1 +$source = Resolve-Path '.\codex-raw-0.1.1-source' +if ((git -C $source rev-parse HEAD).Trim() -ne + '4d814d6e7578a833bb355665862c0f4b83678f89') { + throw 'tag commit mismatch' +} +if ((git -C $source rev-parse ` + 'HEAD:codex-rs/raw-cli/scripts/benchmark_persistent.ps1').Trim() -ne + '2c92994556a50dd2c83be9c844ad41d141a6fbc8') { + throw 'benchmark harness blob mismatch' +} +$harness = Join-Path $source ` + 'codex-rs\raw-cli\scripts\benchmark_persistent.ps1' + +pwsh -File $harness ` + -Exe $exe ` + -Model gpt-5.4-mini ` + -Prompt hello ` + -Runs 10 ` + -TimeoutSeconds 120 +``` + +This command uses the caller's existing Codex Raw browser-login state and +performs 22 live model turns. Do not automate a retry after a partial failure, +because some turns may already have completed. + +## Codex Raw 0.1.0 — Windows x64 + +The first release-bound benchmark was recorded from the immutable Windows x64 +asset for `codex-raw 0.1.0`. One uninterrupted invocation completed exactly +two warm-up turns and twenty measured turns. It performed no retries and had +no failed or timed-out request. + ### Artifact identity | Field | Exact observed value | @@ -35,7 +189,8 @@ no failed or timed-out request. | Target | `x86_64-pc-windows-msvc` | | Benchmark harness | `codex-rs/raw-cli/scripts/benchmark_persistent.ps1` from the tagged commit | | Harness Git blob | `2c92994556a50dd2c83be9c844ad41d141a6fbc8` | -| Harness SHA-256 | `a3b17cd9c9c3445bfbe07c530cb8884f3f9a24f755c772dcc4822368525edc16` | +| Harness blob SHA-256 | `95184d60b7bf753d97d1868263bdb369a3049b3a7f3d6d04127e81e0529e36a1` | +| Harness checkout SHA-256 | `a3b17cd9c9c3445bfbe07c530cb8884f3f9a24f755c772dcc4822368525edc16` (observed CRLF Windows checkout) | The archive digest independently matched both GitHub's asset digest and the release's `SHA256SUMS` record before extraction. @@ -99,7 +254,7 @@ modified. | Successful measured turns | 20 of 20 | | Failed / timed-out turns | 0 / 0 | | Input tokens | exactly 7 in every measured turn | -| Output fingerprints | 19 responses had the same 32-character fingerprint; one app response had a distinct 29-character fingerprint | +| Output fingerprints | 19 responses shared one SHA-256 fingerprint and had 32 output characters; one app response had another fingerprint and 29 output characters | | Output text | deliberately not retained | | Cached-input, output, and reasoning tokens | not retained by the `0.1.0` harness; no values are inferred | @@ -124,10 +279,10 @@ fingerprint of the output. The two warm-up turns completed successfully but their timings were discarded by the tagged `0.1.0` harness before capture; this limitation is recorded in both JSON files. -### Reproduce the aggregate run +### Reproduce the 0.1.0 aggregate run -Download and verify the exact release asset first, then use the harness from -the exact tagged source: +From a clone of this repository, download and verify the exact release asset, +then create a detached worktree for the exact tagged harness: ```powershell $archive = 'codex-raw-0.1.0-x86_64-pc-windows-msvc.zip' @@ -145,7 +300,22 @@ $exe = Resolve-Path ` '.\codex-raw-0.1.0\codex-raw-0.1.0-x86_64-pc-windows-msvc\codex-raw.exe' & $exe --version -pwsh -File .\codex-rs\raw-cli\scripts\benchmark_persistent.ps1 ` +git fetch origin tag codex-raw-v0.1.0 +git worktree add --detach .\codex-raw-0.1.0-source codex-raw-v0.1.0 +$source = Resolve-Path '.\codex-raw-0.1.0-source' +if ((git -C $source rev-parse HEAD).Trim() -ne + '99cc140866c4f66da6d8f283a013e6b43e4458e9') { + throw 'tag commit mismatch' +} +if ((git -C $source rev-parse ` + 'HEAD:codex-rs/raw-cli/scripts/benchmark_persistent.ps1').Trim() -ne + '2c92994556a50dd2c83be9c844ad41d141a6fbc8') { + throw 'benchmark harness blob mismatch' +} +$harness = Join-Path $source ` + 'codex-rs\raw-cli\scripts\benchmark_persistent.ps1' + +pwsh -File $harness ` -Exe $exe ` -Model gpt-5.4-mini ` -Prompt hello ` @@ -169,6 +339,9 @@ because some turns may already have completed. - Publish sanitized per-sample data. Never publish credentials, tokens, account identifiers, private prompts, private infrastructure addresses, absolute local paths, or response text that was not explicitly public. +- Do not publish unsalted output fingerprints for private or low-entropy + outputs; they can reveal the original text. Prefer omitting the fingerprint + unless the output corpus itself is explicitly public. - State unavailable fields as unavailable; do not reconstruct or infer token values that the measured client did not retain. - Comparisons must identify every client artifact and use the same machine, diff --git a/CHANGELOG.md b/CHANGELOG.md index 41a4e459e..640d4eb5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,12 @@ and uses semantic versioning for the `codex-raw` executable. ## [Unreleased] +### Documentation + +- Published a second release-bound benchmark for the immutable `0.1.1` + Windows x64 asset, including sanitized per-sample measurements and exact + provenance. + ## [0.1.1] - 2026-07-29 ### Added diff --git a/benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.provenance.json b/benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.provenance.json new file mode 100644 index 000000000..918486a4b --- /dev/null +++ b/benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.provenance.json @@ -0,0 +1,68 @@ +{ + "schema": "codex-raw-benchmark-provenance-v2", + "status": "completed", + "started_utc": "2026-07-29T05:07:02.1434541+00:00", + "finished_utc": "2026-07-29T05:07:27.2418112+00:00", + "repository": "bproject07/Codex-Source", + "tag": "codex-raw-v0.1.1", + "tag_object": "42336d4b4f7710aa6ec64f41556263d64d0695de", + "tag_commit": "4d814d6e7578a833bb355665862c0f4b83678f89", + "release_id": 361516591, + "release_url": "https://github.com/bproject07/Codex-Source/releases/tag/codex-raw-v0.1.1", + "release_immutable": true, + "release_published_at": "2026-07-29T05:02:57.0000000+00:00", + "asset_name": "codex-raw-0.1.1-x86_64-pc-windows-msvc.zip", + "asset_size_bytes": 12163128, + "asset_sha256": "f883916658fd6b4e1d549dbb50a532c72d9f9efafbdf7dfb7ee129740ff1057d", + "executable_sha256": "0bb4a1d8b2d624d59b7023bc9256cb5643a5c5f9cf46f6b90bf21aeea6026e0c", + "executable_version": "codex-raw 0.1.1", + "target": "x86_64-pc-windows-msvc", + "harness_path": "codex-rs/raw-cli/scripts/benchmark_persistent.ps1", + "harness_git_blob": "2c92994556a50dd2c83be9c844ad41d141a6fbc8", + "harness_blob_sha256": "95184d60b7bf753d97d1868263bdb369a3049b3a7f3d6d04127e81e0529e36a1", + "harness_working_tree_sha256": "a3b17cd9c9c3445bfbe07c530cb8884f3f9a24f755c772dcc4822368525edc16", + "harness_working_tree_line_endings": "CRLF", + "harness_modified": false, + "capture_method": "PowerShell line breakpoint after the measured loop", + "capture_breakpoint_line": 410, + "model": "gpt-5.4-mini", + "prompt": "hello", + "prompt_utf8_sha256": "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824", + "concurrency_per_mode": 1, + "warmup_turns": 2, + "warmup_samples_retained": false, + "measured_turns": 20, + "measured_per_mode": 10, + "retries": 0, + "timeout_seconds": 120, + "run_order": "warm-up app, warm-up API; then paired sequential samples with alternating first mode", + "ttft_definition": "client stopwatch start to first non-empty assistant text delta", + "total_definition": "same stopwatch start to terminal turn/completion event", + "median_method": "arithmetic mean of the two middle sorted values", + "percentile_method": "nearest rank: ceil(p * n) - 1, zero-based", + "samples_file": "benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.samples.json", + "samples_file_sha256": "a56015e18430d0c220d0d2f04fb4d826b1d03c02ea47628fb9225ca2d607e0f5", + "output_text_retained": false, + "output_fingerprint_retained": false, + "wall_clock_interval_consistent_with_measured_samples": false, + "wall_clock_note": "Observed finish minus start is 242.5499 ms shorter than the sum of the sequential measured-turn monotonic durations alone. Wall-clock timestamps are retained only as observed metadata and are not used in latency calculations.", + "os_caption": "Microsoft Windows 11 Pro", + "os_version": "10.0.26200", + "os_build": "26200", + "os_architecture": "X64", + "process_architecture": "X64", + "cpu": [ + { + "name": "12th Gen Intel(R) Core(TM) i7-12700", + "physical_cores": 4, + "logical_cores": 4 + } + ], + "memory_bytes": 12776771584, + "manufacturer": "innotek GmbH", + "model_or_shape": "VirtualBox", + "hypervisor_present": true, + "powershell_version": "7.6.3", + "local_time_zone": "FLE Standard Time", + "network_description": "local workstation connection; route and upstream model placement not controlled" +} diff --git a/benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.samples.json b/benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.samples.json new file mode 100644 index 000000000..f7cfcbe43 --- /dev/null +++ b/benchmarks/data/codex-raw-v0.1.1-windows-x64-20260729T050702Z.samples.json @@ -0,0 +1,189 @@ +{ + "schema": "codex-raw-benchmark-samples-v2", + "warmup_turns_completed": 2, + "warmup_samples_retained": false, + "measured_turns": 20, + "ordering": "one app-server and one api-server per pair; first mode alternates", + "samples": [ + { + "sequence": 1, + "pair": 1, + "mode": "app-server", + "ttft_ms": 1333.8782, + "total_ms": 1563.53, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 2, + "pair": 1, + "mode": "api-server", + "ttft_ms": 1484.2748, + "total_ms": 1693.171, + "input_tokens": 7, + "output_chars": 29 + }, + { + "sequence": 3, + "pair": 2, + "mode": "api-server", + "ttft_ms": 1900.7111, + "total_ms": 2170.6873, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 4, + "pair": 2, + "mode": "app-server", + "ttft_ms": 2390.8814, + "total_ms": 2588.0529, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 5, + "pair": 3, + "mode": "app-server", + "ttft_ms": 683.806, + "total_ms": 932.42, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 6, + "pair": 3, + "mode": "api-server", + "ttft_ms": 898.4213, + "total_ms": 1100.9768, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 7, + "pair": 4, + "mode": "api-server", + "ttft_ms": 633.9082, + "total_ms": 829.794, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 8, + "pair": 4, + "mode": "app-server", + "ttft_ms": 820.8882, + "total_ms": 995.171, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 9, + "pair": 5, + "mode": "app-server", + "ttft_ms": 622.084, + "total_ms": 826.1945, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 10, + "pair": 5, + "mode": "api-server", + "ttft_ms": 671.0134, + "total_ms": 912.4867, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 11, + "pair": 6, + "mode": "api-server", + "ttft_ms": 813.9887, + "total_ms": 1030.7894, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 12, + "pair": 6, + "mode": "app-server", + "ttft_ms": 1050.8383, + "total_ms": 1197.9867, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 13, + "pair": 7, + "mode": "app-server", + "ttft_ms": 675.0878, + "total_ms": 889.8208, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 14, + "pair": 7, + "mode": "api-server", + "ttft_ms": 731.8774, + "total_ms": 919.5329, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 15, + "pair": 8, + "mode": "api-server", + "ttft_ms": 677.7082, + "total_ms": 897.3414, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 16, + "pair": 8, + "mode": "app-server", + "ttft_ms": 810.6472, + "total_ms": 972.288, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 17, + "pair": 9, + "mode": "app-server", + "ttft_ms": 744.8626, + "total_ms": 940.041, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 18, + "pair": 9, + "mode": "api-server", + "ttft_ms": 663.1741, + "total_ms": 866.4232, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 19, + "pair": 10, + "mode": "api-server", + "ttft_ms": 517.0967, + "total_ms": 722.0621, + "input_tokens": 7, + "output_chars": 32 + }, + { + "sequence": 20, + "pair": 10, + "mode": "app-server", + "ttft_ms": 3022.9238, + "total_ms": 3292.1373, + "input_tokens": 7, + "output_chars": 32 + } + ] +}