huggingface/transformers · error · ValueError

--num_tokens_to_generate arguments should be larger than 1

Error message

--num_tokens_to_generate arguments should be larger than 1

What it means

Same FP8 recipe-selection guard, next check: `block_size` is set but not one of the two block layouts DeepGEMM supports — `(128, 128)` or `(1, 128)`. Any other granularity (e.g. `(64, 64)`, `(1, 64)`, `(32, 32)`) is rejected because the kernel recipes are hard-coded for 128-granularity K blocks.

Source

Thrown at benchmark_v2/run_benchmarks.py:98

    handlers = [logging.StreamHandler(sys.stdout)]
    logging.basicConfig(
        level=numeric_level, format="[%(levelname)s - %(asctime)s] %(name)s: %(message)s", handlers=handlers
    )

    logger = logging.getLogger("benchmark_v2")
    logger.info("Starting benchmark discovery and execution")
    logger.info(f"Benchmark run UUID: {benchmark_run_uuid}")
    logger.info(f"Output directory: {args.output_dir}")

    # Error out if one of the arguments is not provided
    if any(arg is None for arg in [args.batch_size, args.sequence_length, args.num_tokens_to_generate]):
        raise ValueError(
            "All of the arguments --batch-size, --sequence-length, and --num-tokens-to-generate are required"
        )

    # We cannot compute ITL if we don't have at least two measurements
    if any(n <= 1 for n in args.num_tokens_to_generate):
        raise ValueError("--num_tokens_to_generate arguments should be larger than 1")

    # If a config file is provided, read it and use the configs therein. They will still be adapted to the given arguments.
    if args.config_file is not None:
        if args.config_file.endswith(".json"):
            with open(args.config_file, "r") as f:
                config_as_dicts = [json.load(f)]
        elif args.config_file.endswith(".jsonl"):
            with open(args.config_file, "r") as f:
                config_as_dicts = [json.loads(line) for line in f if line.startswith("{")]
        else:
            raise ValueError(f"Unsupported config file format: {args.config_file}")
        configs = [BenchmarkConfig.from_dict(config) for config in config_as_dicts]
    else:
        # Otherwise, get the configs for the given coverage level
        configs = get_config_by_level(args.level)

    # Adapt the configs to the given arguments
    configs = adapt_configs(

View on GitHub (pinned to a597f97485)

Solutions

1. Re-quantize the model with `weight_block_size=[128, 128]` (or `[1, 128]`) so it matches DeepGEMM recipes
2. Or use a hub checkpoint already quantized with 128-block FP8 (DeepSeek-style)
3. Otherwise pick a different experts implementation that supports your granularity

Example fix

# before
FbgemmFp8Config(weight_block_size=[64, 64])  # -> ValueError: block_size not in {(128,128),(1,128)}

# after
FbgemmFp8Config(weight_block_size=[128, 128])

block = tuple(model.config.quantization_config.weight_block_size)
assert block in ((128, 128), (1, 128)), f"unsupported block_size {block} for DeepGEMM"

def has_deepgemm_block_size(qcfg) -> bool:
    b = getattr(qcfg, "weight_block_size", None)
    return b is not None and tuple(b) in ((128, 128), (1, 128))

When it happens

Trigger: An FP8 checkpoint quantized with a non-128 `weight_block_size` (say `[64, 64]` or `[16, 16]`) loaded into experts running the `deepgemm` implementation.

Common situations: Custom quantization recipes tuned for other kernels (AWQ/other FP8 schemes with 16/64 blocks); research code exploring finer block sizes then switching dispatch to deepgemm.

Related errors

• All of the arguments --batch-size, --sequence-length, and --
• No benchmark was run successfully
• Unsupported config file format: {args.config_file}
• Incorrect audio source. Must be a valid URL starting with `h
• The audio source is a '{filetype}' file, which librosa canno

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/98cda4172fac34c7. Report an issue: GitHub.