huggingface/transformers · error · ValueError

All of the arguments --batch-size, --sequence-length, and --

Error message

All of the arguments --batch-size, --sequence-length, and --num-tokens-to-generate are required

What it means

`_select_fp8_cast_kwargs` validates the experts' quantization metadata before choosing the DeepGEMM recipe: for FP8 weights it requires a `block_size` (block-wise quantization), but the experts module has `block_size=None`. DeepGEMM only supports block-wise FP8 (granularity 128), so a missing block size means the checkpoint/quantizer config is not block-wise and the recipe cannot be inferred.

Source

Thrown at benchmark_v2/run_benchmarks.py:92

    args = parser.parse_args()

    # Setup logging
    benchmark_run_uuid = str(uuid.uuid4())[:8]
    numeric_level = getattr(logging, args.log_level.upper())

    handlers = [logging.StreamHandler(sys.stdout)]
    logging.basicConfig(
        level=numeric_level, format="[%(levelname)s - %(asctime)s] %(name)s: %(message)s", handlers=handlers
    )

    logger = logging.getLogger("benchmark_v2")
    logger.info("Starting benchmark discovery and execution")
    logger.info(f"Benchmark run UUID: {benchmark_run_uuid}")
    logger.info(f"Output directory: {args.output_dir}")

    # Error out if one of the arguments is not provided
    if any(arg is None for arg in [args.batch_size, args.sequence_length, args.num_tokens_to_generate]):
        raise ValueError(
            "All of the arguments --batch-size, --sequence-length, and --num-tokens-to-generate are required"
        )

    # We cannot compute ITL if we don't have at least two measurements
    if any(n <= 1 for n in args.num_tokens_to_generate):
        raise ValueError("--num_tokens_to_generate arguments should be larger than 1")

    # If a config file is provided, read it and use the configs therein. They will still be adapted to the given arguments.
    if args.config_file is not None:
        if args.config_file.endswith(".json"):
            with open(args.config_file, "r") as f:
                config_as_dicts = [json.load(f)]
        elif args.config_file.endswith(".jsonl"):
            with open(args.config_file, "r") as f:
                config_as_dicts = [json.loads(line) for line in f if line.startswith("{")]
        else:
            raise ValueError(f"Unsupported config file format: {args.config_file}")
        configs = [BenchmarkConfig.from_dict(config) for config in config_as_dicts]

View on GitHub (pinned to a597f97485)

Solutions

1. Use a fine-grained block-wise FP8 checkpoint (quantized with `weight_block_size=[128, 128]` or `[1, 128]`, e.g. DeepSeek-V3/V4 style)
2. If re-quantizing yourself, set the block size in the quantizer config so experts carry `block_size`
3. If the checkpoint is intentionally per-tensor FP8, do not use the deepgemm dispatch — use `grouped_mm` or the default path

Example fix

# before
quant_cfg = FbgemmFp8Config(activation_scheme="dynamic")  # no weight_block_size
# experts get block_size=None -> ValueError

# after
quant_cfg = FbgemmFp8Config(activation_scheme="dynamic", weight_block_size=[128, 128])

qcfg = model.config.quantization_config
block = getattr(qcfg, "weight_block_size", None)
if block is None:
    raise SystemExit("checkpoint is not block-wise FP8; use a [128,128]-block FP8 model for deepgemm")

def is_blockwise_fp8(qcfg) -> bool:
    return getattr(qcfg, "weight_block_size", None) is not None

When it happens

Trigger: Running DeepGEMM experts dispatch on FP8 weights whose `QuantizerConfig`/checkpoint has no `weight_block_size` (e.g. per-tensor or per-channel FP8 quantization like some legacy GPTQ/FP8 merges), so `block_size` arrives as None.

Common situations: Checkpoints quantized with per-tensor FP8 (not fine-grained block FP8); locally re-quantized models where the quantizer dropped `weight_block_size`; mixing DeepSeek-style expectations with generic FP8 checkpoints.

Related errors

• --num_tokens_to_generate arguments should be larger than 1
• No benchmark was run successfully
• Unsupported config file format: {args.config_file}
• Incorrect audio source. Must be a valid URL starting with `h
• The audio source is a '{filetype}' file, which librosa canno

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/7e79836ba6beda27. Report an issue: GitHub.