vllm-project/vllm · error · NotImplementedError

Pipeline parallelism is not supported for this model. Suppor

Error message

Pipeline parallelism is not supported for this model. Supported models implement the `SupportsPP` interface.

What it means

Raised when pipeline_parallel_size > 1 but the model's implementation does not implement the SupportsPP interface (checked via registry.is_pp_supported_model). PP requires the model class to opt in by exposing per-layer information vLLM needs to split stages.

Source

Thrown at vllm/config/model.py:1381

        parallel_config: ParallelConfig,
    ) -> None:
        total_num_attention_heads = self.model_arch_config.total_num_attention_heads
        tensor_parallel_size = parallel_config.tensor_parallel_size
        if total_num_attention_heads % tensor_parallel_size != 0:
            raise ValueError(
                f"Total number of attention heads ({total_num_attention_heads})"
                " must be divisible by tensor parallel size "
                f"({tensor_parallel_size})."
            )

        if parallel_config.enable_expert_parallel:
            self._verify_with_expert_parallelism()

        pipeline_parallel_size = parallel_config.pipeline_parallel_size
        if pipeline_parallel_size > 1 and not self.registry.is_pp_supported_model(
            self.architectures, self
        ):
            raise NotImplementedError(
                "Pipeline parallelism is not supported for this model. "
                "Supported models implement the `SupportsPP` interface."
            )

        decode_context_parallel_size = parallel_config.decode_context_parallel_size
        if decode_context_parallel_size > 1 and not self.use_mla:
            total_num_kv_heads = self.get_total_num_kv_heads()
            if tensor_parallel_size <= total_num_kv_heads:
                raise ValueError(
                    "Decode context parallelism for GQA/MQA requires "
                    f"`--tensor-parallel-size` ({tensor_parallel_size}) to be "
                    "greater than the model's total number of KV heads "
                    f"({total_num_kv_heads}). Increase `--tensor-parallel-size` "
                    "or set `--decode-context-parallel-size 1`."
                )

            max_dcp_size = tensor_parallel_size // total_num_kv_heads
            if decode_context_parallel_size > max_dcp_size:

View on GitHub (pinned to c794754062)

Solutions

1. Set --pipeline-parallel-size 1 and use tensor parallelism (or data parallelism) for scaling instead.
2. Use a checkpoint/architecture that implements SupportsPP (most mainstream LLMs do; check the model class for the interface).
3. For a custom model, implement the SupportsPP interface on the model implementation so the registry recognizes it.

Example fix

# before
vllm serve my-unsupported-model --pipeline-parallel-size 2
# after
vllm serve my-unsupported-model --tensor-parallel-size 2

from vllm.model_executor.models.registry import _MULTIMODAL_MODELS  # pattern only
def pp_supported(registry, architectures, model_config) -> bool:
    return registry.is_pp_supported_model(architectures, model_config)
# gate --pipeline-parallel-size > 1 on this check before engine init

When it happens

Trigger: Launching with --pipeline-parallel-size > 1 for an architecture whose registry entry does not implement SupportsPP; NotImplementedError raised from verify_with_parallel_config.

Common situations: Trying PP to fit a large dense model across nodes and discovering that architecture never got PP support; new/custom model implementations registered without SupportsPP; version differences where PP support was added or removed for a family of models.

Related errors

• cannot use in-process coordinator with bootstrapped transpor
• The quantization method %s is deprecated and will be removed
• Number of experts in the model must be greater than 0 when e
• Total number of attention heads ({total_num_attention_heads}
• The model is an hybrid without a layers_block_type or an att

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/411727a4f51ea1e0. Report an issue: GitHub.