{"record":{"id":"98cda4172fac34c7","repo":"huggingface/transformers","slug":"num-tokens-to-generate-arguments-should-be-large","errorCode":null,"errorMessage":"--num_tokens_to_generate arguments should be larger than 1","messagePattern":"--num_tokens_to_generate arguments should be larger than 1","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"benchmark_v2/run_benchmarks.py","lineNumber":98,"sourceCode":"    handlers = [logging.StreamHandler(sys.stdout)]\n    logging.basicConfig(\n        level=numeric_level, format=\"[%(levelname)s - %(asctime)s] %(name)s: %(message)s\", handlers=handlers\n    )\n\n    logger = logging.getLogger(\"benchmark_v2\")\n    logger.info(\"Starting benchmark discovery and execution\")\n    logger.info(f\"Benchmark run UUID: {benchmark_run_uuid}\")\n    logger.info(f\"Output directory: {args.output_dir}\")\n\n    # Error out if one of the arguments is not provided\n    if any(arg is None for arg in [args.batch_size, args.sequence_length, args.num_tokens_to_generate]):\n        raise ValueError(\n            \"All of the arguments --batch-size, --sequence-length, and --num-tokens-to-generate are required\"\n        )\n\n    # We cannot compute ITL if we don't have at least two measurements\n    if any(n <= 1 for n in args.num_tokens_to_generate):\n        raise ValueError(\"--num_tokens_to_generate arguments should be larger than 1\")\n\n    # If a config file is provided, read it and use the configs therein. They will still be adapted to the given arguments.\n    if args.config_file is not None:\n        if args.config_file.endswith(\".json\"):\n            with open(args.config_file, \"r\") as f:\n                config_as_dicts = [json.load(f)]\n        elif args.config_file.endswith(\".jsonl\"):\n            with open(args.config_file, \"r\") as f:\n                config_as_dicts = [json.loads(line) for line in f if line.startswith(\"{\")]\n        else:\n            raise ValueError(f\"Unsupported config file format: {args.config_file}\")\n        configs = [BenchmarkConfig.from_dict(config) for config in config_as_dicts]\n    else:\n        # Otherwise, get the configs for the given coverage level\n        configs = get_config_by_level(args.level)\n\n    # Adapt the configs to the given arguments\n    configs = adapt_configs(","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/benchmark_v2/run_benchmarks.py#L80-L116","documentation":"Same FP8 recipe-selection guard, next check: `block_size` is set but not one of the two block layouts DeepGEMM supports — `(128, 128)` or `(1, 128)`. Any other granularity (e.g. `(64, 64)`, `(1, 64)`, `(32, 32)`) is rejected because the kernel recipes are hard-coded for 128-granularity K blocks.","triggerScenarios":"An FP8 checkpoint quantized with a non-128 `weight_block_size` (say `[64, 64]` or `[16, 16]`) loaded into experts running the `deepgemm` implementation.","commonSituations":"Custom quantization recipes tuned for other kernels (AWQ/other FP8 schemes with 16/64 blocks); research code exploring finer block sizes then switching dispatch to deepgemm.","solutions":["Re-quantize the model with `weight_block_size=[128, 128]` (or `[1, 128]`) so it matches DeepGEMM recipes","Or use a hub checkpoint already quantized with 128-block FP8 (DeepSeek-style)","Otherwise pick a different experts implementation that supports your granularity"],"exampleFix":"# before\nFbgemmFp8Config(weight_block_size=[64, 64])  # -> ValueError: block_size not in {(128,128),(1,128)}\n\n# after\nFbgemmFp8Config(weight_block_size=[128, 128])","handlingStrategy":"validation","validationCode":"block = tuple(model.config.quantization_config.weight_block_size)\nassert block in ((128, 128), (1, 128)), f\"unsupported block_size {block} for DeepGEMM\"","typeGuard":"def has_deepgemm_block_size(qcfg) -> bool:\n    b = getattr(qcfg, \"weight_block_size\", None)\n    return b is not None and tuple(b) in ((128, 128), (1, 128))","tryCatchPattern":null,"preventionTips":["Standardize on 128-granularity block FP8 when DeepGEMM is in the target stack","Validate quantization config at load time, not at first forward"],"tags":["deepgemm","fp8","block-size","checkpoint-config"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}