huggingface/transformers · error · RuntimeError

Generated {results.size(-1)} tokens, expected {config.num_to

Error message

Generated {results.size(-1)} tokens, expected {config.num_tokens_to_generate}

What it means

`_assert_sm100_requirements` detected an int8-packed (FP4) weight tensor while running on a pre-Blackwell GPU (`is_sm100()` false, i.e. Hopper SM90). DeepGEMM ships no FP4 GEMM kernel for Hopper, so the integration fails loudly instead of silently corrupting output. It raises `NotImplementedError` on purpose: `fp8_linear` treats it as 'DeepGEMM declined' and falls back to Triton.

Source

Thrown at benchmark_v2/framework/benchmark_runner.py:301

            outputs = self.model.generate(**self.inputs, streamer=streamer)

        wall_time_1 = time.perf_counter()
        gpu_metrics = gpu_monitor.stop_and_collect() if gpu_monitor is not None else None

        # Retrieve timestamps and results in a way that allows similar post-processing
        input_tokens = self.inputs["input_ids"].size(-1)
        if config.continuous_batching:
            timestamps = [output.timestamps[:] for output in outputs.values()]
            results = torch.tensor([output.generated_tokens[:] for output in outputs.values()])
        else:
            timestamps = [streamer.timestamps[1:]]  # skip the first timestamp because it's the input tokens
            results = outputs[:, input_tokens:]
        outputs = None
        flush_memory(flush_compile=False)

        # Check if generation had the right number of tokens
        if results.size(-1) != config.num_tokens_to_generate:
            raise RuntimeError(f"Generated {results.size(-1)} tokens, expected {config.num_tokens_to_generate}")

        # Decode outputs
        decoded_output = self.tokenizer.decode(results[0], skip_special_tokens=True)
        shape_and_decoded_output = f"{tuple(results.shape)} | {decoded_output}"

        # Compute metrics
        e2e_latency = wall_time_1 - wall_time_0
        timestamps = torch.tensor(timestamps).sub(wall_time_0).tolist()
        self.logger.info(
            f"Time generate done in {e2e_latency:.2f} seconds. Memory usage: {self.torch_accelerator_module.memory_allocated() / 1024**2:.2f} MB"
        )
        return e2e_latency, timestamps, shape_and_decoded_output, gpu_metrics

    def profile_generate(self, num_tokens_to_profile: int, config_name: str) -> None:
        """Profile the latency of a call to model.generate() with the given (inputs) and (max_new_tokens)."""
        activities = [torch.profiler.ProfilerActivity.CPU]
        if self.device_type == "cuda":
            activities.append(torch.profiler.ProfilerActivity.CUDA)

View on GitHub (pinned to a597f97485)

Solutions

1. Run on a Blackwell (SM100+) GPU — the only arch with an FP4 DeepGEMM kernel
2. Or use an FP8 (block-wise) checkpoint instead of FP4 on Hopper
3. If you hit this via `fp8_linear`, rely on the documented automatic Triton fallback (make sure your call path catches `NotImplementedError` as 'declined')
4. For experts paths, switch dispatch: `model.set_experts_implementation('grouped_mm')`

Example fix

// before
# FP4 (int8-packed) checkpoint on H100
model = AutoModelForCausalLM.from_pretrained("<nvfp4-model>")  # runs on SM90 -> NotImplementedError

// after
# option A: Blackwell node (B200), or option B: FP8 checkpoint + grouped_mm experts
model.set_experts_implementation("grouped_mm")

import torch
from transformers.integrations.deepgemm import is_sm100

fp4_weights = any(p.dtype == torch.int8 for p in model.parameters())
if fp4_weights and not is_sm100():
    model.set_experts_implementation("grouped_mm")  # no FP4 kernel on Hopper

try:
    out = fp8_linear_deepgemm(x, w, w_sf)
except NotImplementedError:
    out = fp8_linear_triton(x, w, w_sf)  # fp8_linear's documented decline->fallback contract

When it happens

Trigger: Loading an FP4-quantized checkpoint (weights stored as int8-packed NVFP4) and running the DeepGEMM linear/experts path on an H100/H200 (SM90). The guard fires before the hub-download + JIT kernel load.

Common situations: Taking an FP4 model authored for B200/B300 nodes and running it on Hopper clusters; mixed fleet where a job lands on the wrong partition; testing FP4 checkpoints locally on older GPUs.

Related errors

• No benchmark was run successfully
• The audio source is a '{filetype}' file, which librosa canno
• No baseline with name '{name}' in {RESULTS_DIR}
• Tensor parallelism was requested, but WORLD_SIZE is not set
• PUSH_TO_HUB_TOKEN is not set, cannot push results to the Hub

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/bd4ca48d72ea9db4. Report an issue: GitHub.