{"record":{"id":"409ff9c2cb23910b","repo":"huggingface/transformers","slug":"tensor-parallelism-was-requested-but-world-size-i","errorCode":null,"errorMessage":"Tensor parallelism was requested, but WORLD_SIZE is not set to more than 1. Launch the benchmark with `torchrun --nproc_per_node=<num_gpus> ...` to run with tensor parallelism.","messagePattern":"Tensor parallelism was requested, but WORLD_SIZE is not set to more than 1\\. Launch the benchmark with `torchrun --nproc_per_node=<num_gpus> \\.\\.\\.` to run with tensor parallelism\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"benchmark_v2/framework/benchmark_config.py","lineNumber":153,"sourceCode":"            logger.error(\n                f\"You have continuous batching and compile enabled, but {self.compile_config.mode = } is not supported.\"\n                \" Supported modes are: default, max-autotune-no-cudagraphs. Changing to default.\"\n            )\n            self.compile_config.mode = \"default\"\n\n    @property\n    def hash(self) -> str:\n        return hashlib.sha256(json.dumps(self.to_dict()).encode()).hexdigest()\n\n    @property\n    def distributed_config(self) -> DistributedConfig | None:\n        \"\"\"Translate `tp_plan` into the `DistributedConfig` that `from_pretrained` expects, or `None` if no TP.\"\"\"\n        if self.tp_plan is None:\n            return None\n        # `torchrun` sets WORLD_SIZE; without it there is no process group to shard over.\n        tp_size = int(os.environ.get(\"WORLD_SIZE\", 1))\n        if tp_size <= 1:\n            raise ValueError(\n                \"Tensor parallelism was requested, but WORLD_SIZE is not set to more than 1. Launch the benchmark \"\n                \"with `torchrun --nproc_per_node=<num_gpus> ...` to run with tensor parallelism.\"\n            )\n        # `DistributedConfig.tp_plan` only takes an explicit plan; leaving it as None makes it use the model's own.\n        return DistributedConfig(tp_size=tp_size, tp_plan=self.tp_plan if isinstance(self.tp_plan, dict) else None)\n\n    def infer_name(self, compact: bool = True) -> str:\n        \"\"\"Infer a human-readable name for the benchmark config, either compact or verbose.\"\"\"\n        if compact:\n            iter_str = f\"w{self.warmup_iterations}_i{self.measurement_iterations}\"\n            gpu_monitor_str = \"monitored\" if self.gpu_monitoring else \"unmonitored\"\n            dimensions_str = f\"b{self.batch_size}_s{self.sequence_length}_n{self.num_tokens_to_generate}\"\n            attn_code = self.attn_implementation\n            compile_str = f\"compiled_{self.compile_config.mode}\" if self.compile_config is not None else \"uncompiled\"\n            kernelize_str = \"kernelized\" if self.kernelize else \"unkernelized\"\n            continuous_batching_str = \"cb\" if self.continuous_batching else \"generate\"\n            tp_str = \"tp\" if self.tp_plan is not None else \"no_tp\"\n            sep = \"-\"","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/benchmark_v2/framework/benchmark_config.py#L135-L171","documentation":"The `kernels-community/deep-gemm` module did load, but attribute probing (`getattr`/`resolve_internal_import`) found that one or more required symbols are missing — e.g. `fp8_fp4_gemm_nt`, `m_grouped_*_gemm_*`, `utils.per_token_cast_to_fp8`, `transform_sf_into_required_layout`, `transform_weights_for_mega_moe`, `get_symm_buffer_for_mega_moe`, `get_mk_alignment_for_contiguous_layout`, `fp8_fp4_mega_moe`. This indicates the installed `kernels` package version is incompatible (older kernel revision without the newer symbols). The message embeds the required version range (`KERNELS_MIN_VERSION <= version < KERNELS_MAX_VERSION`).","triggerScenarios":"Loading the DeepGEMM path with an old/newer `kernels` package than the integration was written against — e.g. `pip install kernels==0.1.x` predating `fp8_fp4_mega_moe`, or a too-new major whose API renamed symbols. First DeepGEMM forward triggers `load_deepgemm_kernel()` which raises this ImportError.","commonSituations":"Pinned/older `kernels` in a shared Docker image; transformers upgraded (expecting new deep-gemm symbols like mega-MoE) while `kernels` stayed behind; bleeding-edge `kernels` 2.x with breaking renames.","solutions":["Install the explicitly suggested compatible version: `pip install kernels==<KERNELS_MIN_VERSION from the message>`","Check `pip show kernels` and align it with the range stated in the error text","If you cannot change the env, avoid the DeepGEMM path (`set_experts_implementation('grouped_mm')` or standard FP8 Triton linear)","Report/verify in the transformers changelog which `kernels` range matches your transformers version"],"exampleFix":"// before\n# kernels==0.1.4 installed -> ImportError: missing symbols fp8_fp4_mega_moe, ...\n\n// after\npip install \"kernels==0.2.*\"  # whatever satisfies the range printed in the error\n# or opt out of DeepGEMM:\nmodel.set_experts_implementation(\"grouped_mm\")","handlingStrategy":"validation","validationCode":"import kernels\nfrom packaging.version import Version\n\nv = Version(kernels.__version__)\nassert KERNELS_MIN_VERSION <= v < KERNELS_MAX_VERSION, f\"kernels {v} out of supported range\"","typeGuard":null,"tryCatchPattern":"try:\n    deepgemm = load_deepgemm_kernel()\nexcept ImportError as e:\n    if \"missing required symbols\" in str(e):\n        subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", f\"kernels=={KERNELS_MIN_VERSION}\"])\n        deepgemm = load_deepgemm_kernel()\n    else:\n        raise","preventionTips":["Lock `kernels` to the version range your transformers version documents","Upgrade transformers and kernels together in lockstep","Smoke-test load_deepgemm_kernel() in CI on the target torch/CUDA image"],"tags":["deepgemm","kernels","version-conflict","installation"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}