{"record":{"id":"c44308ad33b260bd","repo":"huggingface/transformers","slug":"block-size-must-be-at-least-but-got","errorCode":null,"errorMessage":"Block size must be at least {}, but got {}","messagePattern":"Block size must be at least (.+?), but got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/continuous_batching/cache.py","lineNumber":179,"sourceCode":"            config: Model configuration\n            continuous_batching_config: Continuous batching configuration containing cache parameters\n            device: Device for the cache tensors\n            distributed_helper: TP-aware helper. Used to dispatch attention heads and ensure coherent cache size\n            tp_plan: Tensor parallelism plan\n            dtype: Data type of the activation and the cache (for now, these are the same)\n        \"\"\"\n        self.config = config\n        self.dtype = dtype\n        self.device = device\n\n        # Extract model dimensions\n        self.num_key_value_heads: int = find_num_kv_heads(config)\n        self.head_dim: int = find_head_dim(config)\n\n        # Extract cache dimensions. Default used to be 32, now it's 256 to be compatible with flash_with_kvcache.\n        self.block_size = continuous_batching_config.block_size\n        if self.block_size < self._min_block_size:\n            raise ValueError(f\"Block size must be at least {self._min_block_size}, but got {self.block_size}\")\n\n        # Group layers depending on the attention mix\n        layer_groups, group_types = group_layers_by_attn_type(config)\n        group_size = len(layer_groups[0])\n        self.num_groups = len(layer_groups)\n\n        self.sliding_windows = {}\n        self.layer_index_to_group_indices = {}\n        for i, group in enumerate(layer_groups):\n            sliding_window = config.sliding_window if group_types[i] == \"sliding_attention\" else 1\n            for j, layer in enumerate(group):\n                self.layer_index_to_group_indices[layer] = (i, j)\n                self.sliding_windows[layer] = sliding_window\n\n        # Check if the KV heads are part of the TP plan. If they are not, the cache does not need plan for TP.\n        # TODO: this is fragile. If your model fails to TP properly because of this, please open an issue.\n        kv_is_tp = True\n        for key in [\"layers.*.self_attn.k_proj\", \"layers.*.self_attn.v_proj\"]:","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/cache.py#L161-L197","documentation":"ValueError raised while building the paged KV cache: continuous_batching_config.block_size must be at least Cache._min_block_size. The default was raised from 32 to 256 tokens to stay compatible with flash_attn_with_kvcache; smaller blocks break flash-attention's paged KV kernel constraints.","triggerScenarios":"Constructing the continuous-batching cache with ContinuousBatchingConfig(block_size=16 or 32); copying vLLM-style block sizes (16) into transformers' implementation; stale configs from before the 256 default.","commonSituations":"Porting vLLM tuning guides (which use block_size 16) to transformers continuous batching; trying to reduce memory fragmentation with small blocks.","solutions":["Use block_size >= the class's minimum (check Cache._min_block_size; historically 256 for flash-attn compatibility)","Or omit block_size to accept the library default","If you truly need small blocks, you cannot with this path — use a different cache implementation"],"exampleFix":"# before\ncb_cfg = ContinuousBatchingConfig(block_size=16)\n# after\ncb_cfg = ContinuousBatchingConfig(block_size=256)","handlingStrategy":"validation","validationCode":"from transformers.generation.continuous_batching.cache import Cache\ncb_cfg.block_size = max(cb_cfg.block_size or 0, Cache._min_block_size)","typeGuard":"def valid_block_size(bs: int) -> bool:\n    from transformers.generation.continuous_batching.cache import Cache\n    return bs >= Cache._min_block_size","tryCatchPattern":null,"preventionTips":["Do not copy vLLM's block_size=16 into transformers continuous batching","Omit block_size to take the library default (256)"],"tags":["continuous-batching","kv-cache","block-size","flash-attention"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}