huggingface/transformers · error · ValueError

Tensor parallelism was requested, but WORLD_SIZE is not set

Error message

Tensor parallelism was requested, but WORLD_SIZE is not set to more than 1. Launch the benchmark with `torchrun --nproc_per_node=<num_gpus> ...` to run with tensor parallelism.

What it means

The `kernels-community/deep-gemm` module did load, but attribute probing (`getattr`/`resolve_internal_import`) found that one or more required symbols are missing — e.g. `fp8_fp4_gemm_nt`, `m_grouped_*_gemm_*`, `utils.per_token_cast_to_fp8`, `transform_sf_into_required_layout`, `transform_weights_for_mega_moe`, `get_symm_buffer_for_mega_moe`, `get_mk_alignment_for_contiguous_layout`, `fp8_fp4_mega_moe`. This indicates the installed `kernels` package version is incompatible (older kernel revision without the newer symbols). The message embeds the required version range (`KERNELS_MIN_VERSION <= version < KERNELS_MAX_VERSION`).

Source

Thrown at benchmark_v2/framework/benchmark_config.py:153

            logger.error(
                f"You have continuous batching and compile enabled, but {self.compile_config.mode = } is not supported."
                " Supported modes are: default, max-autotune-no-cudagraphs. Changing to default."
            )
            self.compile_config.mode = "default"

    @property
    def hash(self) -> str:
        return hashlib.sha256(json.dumps(self.to_dict()).encode()).hexdigest()

    @property
    def distributed_config(self) -> DistributedConfig | None:
        """Translate `tp_plan` into the `DistributedConfig` that `from_pretrained` expects, or `None` if no TP."""
        if self.tp_plan is None:
            return None
        # `torchrun` sets WORLD_SIZE; without it there is no process group to shard over.
        tp_size = int(os.environ.get("WORLD_SIZE", 1))
        if tp_size <= 1:
            raise ValueError(
                "Tensor parallelism was requested, but WORLD_SIZE is not set to more than 1. Launch the benchmark "
                "with `torchrun --nproc_per_node=<num_gpus> ...` to run with tensor parallelism."
            )
        # `DistributedConfig.tp_plan` only takes an explicit plan; leaving it as None makes it use the model's own.
        return DistributedConfig(tp_size=tp_size, tp_plan=self.tp_plan if isinstance(self.tp_plan, dict) else None)

    def infer_name(self, compact: bool = True) -> str:
        """Infer a human-readable name for the benchmark config, either compact or verbose."""
        if compact:
            iter_str = f"w{self.warmup_iterations}_i{self.measurement_iterations}"
            gpu_monitor_str = "monitored" if self.gpu_monitoring else "unmonitored"
            dimensions_str = f"b{self.batch_size}_s{self.sequence_length}_n{self.num_tokens_to_generate}"
            attn_code = self.attn_implementation
            compile_str = f"compiled_{self.compile_config.mode}" if self.compile_config is not None else "uncompiled"
            kernelize_str = "kernelized" if self.kernelize else "unkernelized"
            continuous_batching_str = "cb" if self.continuous_batching else "generate"
            tp_str = "tp" if self.tp_plan is not None else "no_tp"
            sep = "-"

View on GitHub (pinned to a597f97485)

Solutions

1. Install the explicitly suggested compatible version: `pip install kernels==<KERNELS_MIN_VERSION from the message>`
2. Check `pip show kernels` and align it with the range stated in the error text
3. If you cannot change the env, avoid the DeepGEMM path (`set_experts_implementation('grouped_mm')` or standard FP8 Triton linear)
4. Report/verify in the transformers changelog which `kernels` range matches your transformers version

Example fix

// before
# kernels==0.1.4 installed -> ImportError: missing symbols fp8_fp4_mega_moe, ...

// after
pip install "kernels==0.2.*"  # whatever satisfies the range printed in the error
# or opt out of DeepGEMM:
model.set_experts_implementation("grouped_mm")

import kernels
from packaging.version import Version

v = Version(kernels.__version__)
assert KERNELS_MIN_VERSION <= v < KERNELS_MAX_VERSION, f"kernels {v} out of supported range"

try:
    deepgemm = load_deepgemm_kernel()
except ImportError as e:
    if "missing required symbols" in str(e):
        subprocess.check_call([sys.executable, "-m", "pip", "install", f"kernels=={KERNELS_MIN_VERSION}"])
        deepgemm = load_deepgemm_kernel()
    else:
        raise

When it happens

Trigger: Loading the DeepGEMM path with an old/newer `kernels` package than the integration was written against — e.g. `pip install kernels==0.1.x` predating `fp8_fp4_mega_moe`, or a too-new major whose API renamed symbols. First DeepGEMM forward triggers `load_deepgemm_kernel()` which raises this ImportError.

Common situations: Pinned/older `kernels` in a shared Docker image; transformers upgraded (expecting new deep-gemm symbols like mega-MoE) while `kernels` stayed behind; bleeding-edge `kernels` 2.x with breaking renames.

Related errors

• No baseline with name '{name}' in {RESULTS_DIR}
• TP and DP cannot be used together
• Generated {results.size(-1)} tokens, expected {config.num_to
• No benchmark was run successfully
• PUSH_TO_HUB_TOKEN is not set, cannot push results to the Hub

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/409ff9c2cb23910b. Report an issue: GitHub.