Skip to content

[Diffusion]Centralize prompt normalization and strip null negative prompts#3780

Open
maang-h wants to merge 2 commits into
vllm-project:mainfrom
maang-h:refactor/diffusion-prompt-utils-pr1
Open

[Diffusion]Centralize prompt normalization and strip null negative prompts#3780
maang-h wants to merge 2 commits into
vllm-project:mainfrom
maang-h:refactor/diffusion-prompt-utils-pr1

Conversation

@maang-h
Copy link
Copy Markdown

@maang-h maang-h commented May 20, 2026

Purpose

Fix inconsistent diffusion prompt handling in online serving where serving_chat always injected negative_prompt: None into OmniTextPrompt, even when the client did not provide a negative prompt. Downstream pipelines then had to duplicate ad-hoc parsing logic, and SD3 could treat missing negatives as empty strings and unintentionally enable CFG-style behavior.

This PR (PR1 of a two-part cleanup):

  • Adds shared helpers in vllm_omni/diffusion/prompt_utils.py to normalize prompt entries (strip None/blank negative_prompt keys while preserving multimodal fields) and to extract batched (prompt, negative_prompt) pairs.
  • Normalizes OmniDiffusionRequest.prompts in __post_init__ and uses has_negative_prompt() for do_classifier_free_guidance.
  • Updates serving_chat diffusion paths to only set negative_prompt when it is non-None (aligned with api_server image generation).
  • Migrates Flux, SD3, and Qwen-Image pipelines to extract_batch_prompts() and removes duplicated TODO/hack blocks.

Test Plan

New unit tests:

  • pytest tests/diffusion/test_prompt_utils.py -v

Test Result

PASS

@chatgpt-codex-connector
Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

Copy link
Copy Markdown
Collaborator

@RuixiangMa RuixiangMa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maang-h Thanks for the contribution! Leaving a fe comments

Comment thread vllm_omni/diffusion/prompt_utils.py
Comment thread tests/diffusion/test_prompt_utils.py
Comment thread vllm_omni/diffusion/prompt_utils.py
Signed-off-by: maang <maang_h@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants