{"record":{"id":"bd4ca48d72ea9db4","repo":"huggingface/transformers","slug":"generated-results-size-1-tokens-expected-con","errorCode":null,"errorMessage":"Generated {results.size(-1)} tokens, expected {config.num_tokens_to_generate}","messagePattern":"Generated (.+?) tokens, expected (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"benchmark_v2/framework/benchmark_runner.py","lineNumber":301,"sourceCode":"            outputs = self.model.generate(**self.inputs, streamer=streamer)\n\n        wall_time_1 = time.perf_counter()\n        gpu_metrics = gpu_monitor.stop_and_collect() if gpu_monitor is not None else None\n\n        # Retrieve timestamps and results in a way that allows similar post-processing\n        input_tokens = self.inputs[\"input_ids\"].size(-1)\n        if config.continuous_batching:\n            timestamps = [output.timestamps[:] for output in outputs.values()]\n            results = torch.tensor([output.generated_tokens[:] for output in outputs.values()])\n        else:\n            timestamps = [streamer.timestamps[1:]]  # skip the first timestamp because it's the input tokens\n            results = outputs[:, input_tokens:]\n        outputs = None\n        flush_memory(flush_compile=False)\n\n        # Check if generation had the right number of tokens\n        if results.size(-1) != config.num_tokens_to_generate:\n            raise RuntimeError(f\"Generated {results.size(-1)} tokens, expected {config.num_tokens_to_generate}\")\n\n        # Decode outputs\n        decoded_output = self.tokenizer.decode(results[0], skip_special_tokens=True)\n        shape_and_decoded_output = f\"{tuple(results.shape)} | {decoded_output}\"\n\n        # Compute metrics\n        e2e_latency = wall_time_1 - wall_time_0\n        timestamps = torch.tensor(timestamps).sub(wall_time_0).tolist()\n        self.logger.info(\n            f\"Time generate done in {e2e_latency:.2f} seconds. Memory usage: {self.torch_accelerator_module.memory_allocated() / 1024**2:.2f} MB\"\n        )\n        return e2e_latency, timestamps, shape_and_decoded_output, gpu_metrics\n\n    def profile_generate(self, num_tokens_to_profile: int, config_name: str) -> None:\n        \"\"\"Profile the latency of a call to model.generate() with the given (inputs) and (max_new_tokens).\"\"\"\n        activities = [torch.profiler.ProfilerActivity.CPU]\n        if self.device_type == \"cuda\":\n            activities.append(torch.profiler.ProfilerActivity.CUDA)","sourceCodeStart":283,"sourceCodeEnd":319,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/benchmark_v2/framework/benchmark_runner.py#L283-L319","documentation":"`_assert_sm100_requirements` detected an int8-packed (FP4) weight tensor while running on a pre-Blackwell GPU (`is_sm100()` false, i.e. Hopper SM90). DeepGEMM ships no FP4 GEMM kernel for Hopper, so the integration fails loudly instead of silently corrupting output. It raises `NotImplementedError` on purpose: `fp8_linear` treats it as 'DeepGEMM declined' and falls back to Triton.","triggerScenarios":"Loading an FP4-quantized checkpoint (weights stored as int8-packed NVFP4) and running the DeepGEMM linear/experts path on an H100/H200 (SM90). The guard fires before the hub-download + JIT kernel load.","commonSituations":"Taking an FP4 model authored for B200/B300 nodes and running it on Hopper clusters; mixed fleet where a job lands on the wrong partition; testing FP4 checkpoints locally on older GPUs.","solutions":["Run on a Blackwell (SM100+) GPU — the only arch with an FP4 DeepGEMM kernel","Or use an FP8 (block-wise) checkpoint instead of FP4 on Hopper","If you hit this via `fp8_linear`, rely on the documented automatic Triton fallback (make sure your call path catches `NotImplementedError` as 'declined')","For experts paths, switch dispatch: `model.set_experts_implementation('grouped_mm')`"],"exampleFix":"// before\n# FP4 (int8-packed) checkpoint on H100\nmodel = AutoModelForCausalLM.from_pretrained(\"<nvfp4-model>\")  # runs on SM90 -> NotImplementedError\n\n// after\n# option A: Blackwell node (B200), or option B: FP8 checkpoint + grouped_mm experts\nmodel.set_experts_implementation(\"grouped_mm\")","handlingStrategy":"fallback","validationCode":"import torch\nfrom transformers.integrations.deepgemm import is_sm100\n\nfp4_weights = any(p.dtype == torch.int8 for p in model.parameters())\nif fp4_weights and not is_sm100():\n    model.set_experts_implementation(\"grouped_mm\")  # no FP4 kernel on Hopper","typeGuard":null,"tryCatchPattern":"try:\n    out = fp8_linear_deepgemm(x, w, w_sf)\nexcept NotImplementedError:\n    out = fp8_linear_triton(x, w, w_sf)  # fp8_linear's documented decline->fallback contract","preventionTips":["Check `torch.cuda.get_device_capability() >= (10, 0)` before selecting FP4 checkpoints","Keep an FP8 variant of the model available for Hopper nodes","Treat NotImplementedError from DeepGEMM guards as 'declined', never as fatal"],"tags":["deepgemm","fp4","gpu-architecture","hopper","blackwell"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}