{"record":{"id":"7e79836ba6beda27","repo":"huggingface/transformers","slug":"all-of-the-arguments-batch-size-sequence-leng","errorCode":null,"errorMessage":"All of the arguments --batch-size, --sequence-length, and --num-tokens-to-generate are required","messagePattern":"All of the arguments --batch-size, --sequence-length, and --num-tokens-to-generate are required","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"benchmark_v2/run_benchmarks.py","lineNumber":92,"sourceCode":"    args = parser.parse_args()\n\n    # Setup logging\n    benchmark_run_uuid = str(uuid.uuid4())[:8]\n    numeric_level = getattr(logging, args.log_level.upper())\n\n    handlers = [logging.StreamHandler(sys.stdout)]\n    logging.basicConfig(\n        level=numeric_level, format=\"[%(levelname)s - %(asctime)s] %(name)s: %(message)s\", handlers=handlers\n    )\n\n    logger = logging.getLogger(\"benchmark_v2\")\n    logger.info(\"Starting benchmark discovery and execution\")\n    logger.info(f\"Benchmark run UUID: {benchmark_run_uuid}\")\n    logger.info(f\"Output directory: {args.output_dir}\")\n\n    # Error out if one of the arguments is not provided\n    if any(arg is None for arg in [args.batch_size, args.sequence_length, args.num_tokens_to_generate]):\n        raise ValueError(\n            \"All of the arguments --batch-size, --sequence-length, and --num-tokens-to-generate are required\"\n        )\n\n    # We cannot compute ITL if we don't have at least two measurements\n    if any(n <= 1 for n in args.num_tokens_to_generate):\n        raise ValueError(\"--num_tokens_to_generate arguments should be larger than 1\")\n\n    # If a config file is provided, read it and use the configs therein. They will still be adapted to the given arguments.\n    if args.config_file is not None:\n        if args.config_file.endswith(\".json\"):\n            with open(args.config_file, \"r\") as f:\n                config_as_dicts = [json.load(f)]\n        elif args.config_file.endswith(\".jsonl\"):\n            with open(args.config_file, \"r\") as f:\n                config_as_dicts = [json.loads(line) for line in f if line.startswith(\"{\")]\n        else:\n            raise ValueError(f\"Unsupported config file format: {args.config_file}\")\n        configs = [BenchmarkConfig.from_dict(config) for config in config_as_dicts]","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/benchmark_v2/run_benchmarks.py#L74-L110","documentation":"`_select_fp8_cast_kwargs` validates the experts' quantization metadata before choosing the DeepGEMM recipe: for FP8 weights it requires a `block_size` (block-wise quantization), but the experts module has `block_size=None`. DeepGEMM only supports block-wise FP8 (granularity 128), so a missing block size means the checkpoint/quantizer config is not block-wise and the recipe cannot be inferred.","triggerScenarios":"Running DeepGEMM experts dispatch on FP8 weights whose `QuantizerConfig`/checkpoint has no `weight_block_size` (e.g. per-tensor or per-channel FP8 quantization like some legacy GPTQ/FP8 merges), so `block_size` arrives as None.","commonSituations":"Checkpoints quantized with per-tensor FP8 (not fine-grained block FP8); locally re-quantized models where the quantizer dropped `weight_block_size`; mixing DeepSeek-style expectations with generic FP8 checkpoints.","solutions":["Use a fine-grained block-wise FP8 checkpoint (quantized with `weight_block_size=[128, 128]` or `[1, 128]`, e.g. DeepSeek-V3/V4 style)","If re-quantizing yourself, set the block size in the quantizer config so experts carry `block_size`","If the checkpoint is intentionally per-tensor FP8, do not use the deepgemm dispatch — use `grouped_mm` or the default path"],"exampleFix":"# before\nquant_cfg = FbgemmFp8Config(activation_scheme=\"dynamic\")  # no weight_block_size\n# experts get block_size=None -> ValueError\n\n# after\nquant_cfg = FbgemmFp8Config(activation_scheme=\"dynamic\", weight_block_size=[128, 128])","handlingStrategy":"validation","validationCode":"qcfg = model.config.quantization_config\nblock = getattr(qcfg, \"weight_block_size\", None)\nif block is None:\n    raise SystemExit(\"checkpoint is not block-wise FP8; use a [128,128]-block FP8 model for deepgemm\")","typeGuard":"def is_blockwise_fp8(qcfg) -> bool:\n    return getattr(qcfg, \"weight_block_size\", None) is not None","tryCatchPattern":null,"preventionTips":["Only pair deepgemm experts with fine-grained FP8 checkpoints (weight_block_size set)","Record quantization scheme in model cards and validate it in your loading script","Default to grouped_mm for unknown checkpoints, opt into deepgemm only after checks"],"tags":["deepgemm","fp8","block-quantization","checkpoint-config"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}