Skip to content

[WIP][CI][Accuracy] Add HunyuanImage3 pixel accuracy test and nightly CI#3790

Open
BLANKETusers wants to merge 12 commits into
vllm-project:mainfrom
BLANKETusers:pr-hunyuan-accuracy-fix
Open

[WIP][CI][Accuracy] Add HunyuanImage3 pixel accuracy test and nightly CI#3790
BLANKETusers wants to merge 12 commits into
vllm-project:mainfrom
BLANKETusers:pr-hunyuan-accuracy-fix

Conversation

@BLANKETusers
Copy link
Copy Markdown
Contributor

Summary

  • Add assert_images_pixel_close helper for full-image pixel-level comparison
    with mean/p99 absolute channel difference metrics and detailed diagnostics
  • Add test_hunyuan_image3_pixel_accuracy that generates images via offline
    end2end.py and compares output against a pre-saved baseline image
  • Add nightly CI step (4× H100) in the Diffusion X2I group to gate pixel
    accuracy regressions
  • Rename diffusers_imagebaseline_image across accuracy helper APIs
    (assert_similarity, assert_image_sequence_similarity)

Files changed

File Change
.buildkite/test-nightly.yml +38 lines: new CI step vllm-omni · HunyuanImage3 · Accuracy Test
tests/assets/hunyuan/hunyuan_baseline.png Baseline reference image (1024×1024)
tests/e2e/accuracy/helpers.py +68 lines: assert_images_pixel_close; rename params
tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py +142 lines: new test

Test plan

HUNYUAN_IMAGE3_DEPLOY_CONFIG=../hunyuan_image3_dit_copy.yaml pytest -s -v tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py --run-level

Test Result

1 passed, 18 warnings in 101.49s (0:01:41)

Pixel Metrics

tencent/HunyuanImage-3.0-Instruct — (offline vs baseline)

Metric Value Threshold Status
mean_abs_diff 0.000000 ≤ 0.02
p99_abs_diff 0.000000 ≤ 0.10
p50 0.000000
p90 0.000000
p95 0.000000
p99 0.000000
p99.9 0.000000

Mismatch ratios (pixel_ratio / channel_ratio)

Threshold (1/255) 0 1 2 4 8 16 32 64 128
pixel_ratio 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
channel_ratio 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Signed-off-by: zzh <943967662@qq.com>
Signed-off-by: zzh <943967662@qq.com>
Signed-off-by: zzh <943967662@qq.com>
Signed-off-by: zzh <943967662@qq.com>
… baseline

Signed-off-by: zzh <943967662@qq.com>
Signed-off-by: zzh <943967662@qq.com>
…eline_image comparison

Signed-off-by: zzh <943967662@qq.com>
Signed-off-by: zzh <943967662@qq.com>
Signed-off-by: zzh <943967662@qq.com>
Signed-off-by: zzh <943967662@qq.com>
Signed-off-by: zzh <943967662@qq.com>
@chatgpt-codex-connector
Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@BLANKETusers BLANKETusers changed the title Pr hunyuan accuracy fix [WIP][CI][Accuracy] Add HunyuanImage3 pixel accuracy test and nightly CI May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant