vllm-project/vllm · error

Image generation should not fail

Error message

Image generation should not fail

What it means

The PyNCCL EPLB communicator uses NCCL collectives for expert-weight redistribution, which only work on CUDA-like (GPU) tensors. The expert weight tensors are on CPU, so this backend cannot be constructed.

Source

Thrown at rust/src/bench/src/datasets/random_mm.rs:443

            Ok(text)
        })
        .collect();
    let prompts = prompts?;

    // Generate images for each request (parallel per request)
    // Each request gets its own RNG seeded deterministically.
    let rid_prefix = request_id_prefix.to_string();
    let result: Vec<SampleRequest> = prompts
        .into_par_iter()
        .enumerate()
        .map(|(i, prompt)| {
            let mut item_rng =
                StdRng::seed_from_u64(seed.wrapping_add(i as u64).wrapping_add(0xBEEF));
            let mm_items: Vec<Arc<str>> = mm_configs[i]
                .iter()
                .map(|key| {
                    generate_random_image(key.width, key.height, &mut item_rng)
                        .expect("Image generation should not fail")
                })
                .collect();

            let mm_content: Option<Arc<[Arc<str>]>> = if mm_items.is_empty() {
                None
            } else {
                Some(Arc::from(mm_items))
            };

            // --enable-multimodal-chat: pre-build the full chat `messages` array
            // (text part + mm items) at dataset time, mirroring Python's
            // apply_multimodal_chat_transformation. mm content moves inside the
            // messages string; the backend splices it verbatim.
            let (mm_content, chat_messages_json) = if enable_multimodal_chat {
                let msgs = build_chat_messages_json(&prompt, mm_content.as_deref());
                (None, Some(Arc::from(msgs.as_str())))
            } else {
                (mm_content, None)

View on GitHub (pinned to c794754062)

Solutions

1. Use the 'torch_gloo' backend for CPU expert weights (it is built on the CPU gloo group)
2. Move expert weights to GPU before creating the pynccl EPLB communicator

Example fix

# before
comm = create_eplb_communicator(..., backend="pynccl")  # weights on CPU

# after
comm = create_eplb_communicator(..., backend="torch_gloo")

device_type = expert_weights[0][0].device.type if expert_weights and expert_weights[0] else 'cpu'
backend = 'pynccl' if device_type == 'cuda' else 'torch_gloo'

def weights_on_cuda_like(expert_weights) -> bool:
    return bool(expert_weights) and expert_weights[0][0].device.type.startswith('cuda')

When it happens

Trigger: Requesting backend='pynccl' when expert weights live on CPU, e.g. CPU offloading of MoE weights or a stateless/meta-device initialization phase where tensors are still on CPU.

Common situations: CPU MoE inference setups; initialization order where the communicator factory runs before weights are moved to GPU; test harnesses constructing the communicator with CPU tensors.

Related errors

• cannot continue the final message when the last message is n
• chat request must contain at least one message
• chat template is required but none was configured
• unsupported multimodal content: {0}
• {self.communicator} communicator is incompatible with async

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/ed03572dd325d1c1. Report an issue: GitHub.